vSAN Cluster fails to come up after manual shutdown using reboot_helper.py script
search cancel

vSAN Cluster fails to come up after manual shutdown using reboot_helper.py script

book

Article ID: 430304

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

  • The vSAN cluster fails to restart after a manual shutdown using the prepare script documented in Manually Shut Down and Restart the vSAN Cluster

  • When executing the recover script to restart the vSAN cluster it fails with the below error:

    Begin to recover the cluster...
    Time among connected hosts are synchronized.
    ERROR:root:Fail to get the value of the option 'https_tunnel' fromconf file /etc/vmware/vsan/vsanperf.conf
    Traceback (most recent call last):
      File "/lib64/python3.11/configparser.py", line 805, in get
      File "/lib64/python3.11/collections/__init__.py", line 1012, in __getitem__
      File "/lib64/python3.11/collections/__init__.py", line 1004, in __missing__
    KeyError: 'https_tunnel'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/vmware/vsan/perfsvc/cliutils.py", line 90, in LoadUseHTTPSTunnelConfig
      File "/lib64/python3.11/configparser.py", line 844, in getboolean
      File "/lib64/python3.11/configparser.py", line 824, in _get_conv
      File "/lib64/python3.11/configparser.py", line 819, in _get
      File "/lib64/python3.11/configparser.py", line 808, in get
    configparser.NoOptionError: No option 'https_tunnel' in section: 'VSANPERF'
    Exiting cluster recovery.
    Encounter error when schedule vSAN reconfig task. Please check vSAN log for more detail and retry.

  • From vsanmgmt.log the below backtrace is seen:
    2026-02-21T16:33:38.106Z In(14) vsand[2112712]: [opID=MainThread statsdaemon::_preCheckForEnablingHighResolutionMode] _preCheckForEnablingHighResolutionMode, preCheckPassed => False, isVsanEnabled => False, self.perfEnabled => True, isWitnessHost => False, isComputerOnlyHost => False, isStatsObjectExisting => False, currentObjSize => 0
    2026-02-21T16:33:38.106Z In(14) vsand[2112712]: [opID=MainThread VsanPyVmomiProfiler::log] Profiler:
    2026-02-21T16:33:38.106Z In(14) vsand[2112712]: [opID=MainThread VsanPyVmomiProfiler::logProfile]   CheckAndHandleHighResolutionMode_bypassCounter_False: 0.00s, consumed: 99436KB (+0KB), consumedPeak: 99436KB (+0KB), effectiveMin: 174080KB (+0KB), effectiveMinPeak: 174080KB (+0KB), implicitMin: 126508KB (+0KB), requestedMinPeak: 126732KB (+0KB)
    2026-02-21T16:33:43.831Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:33:43.834Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:33:53.842Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:33:53.845Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:03.852Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:03.856Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:13.862Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:13.866Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:23.873Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:23.876Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:33.883Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:33.887Z In(14) vsand[2112793]: [opID=VsanHostEventsProcessorImplEventsInit VsanSystemImpl::_ConfigInfoPrintLog] configInfo: skipped
    2026-02-21T16:34:37.764Z Er(11) vsand[2112761]: [opID=2112761 VsanTraceObjectManager::backupTraceFiles] Fail to back up trace: Operation not allowed because the VMKernel is shutting down
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]: Traceback (most recent call last):
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]:   File "/usr/lib/vmware/vsan/perfsvc/VsanTraceObjectManager.py", line 710, in backupTraceFiles
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]:   File "/usr/lib/vmware/vsan/perfsvc/VsanTraceObjectManager.py", line 702, in checkBackupRequirements
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]:   File "/usr/lib/vmware/vsan/perfsvc/VsanTraceObjectManager.py", line 220, in exists
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]:   File "/usr/lib/vmware/vsan/perfsvc/VsanTraceObjectManager.py", line 196, in getPath
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]:   File "/usr/lib/vmware/vsan/perfsvc/VsanHostHelper.py", line 1421, in GetVsanDatastorePath
    2026-02-21T16:34:37.764Z Er(11)[+] vsand[2112761]: RuntimeError: Operation not allowed because the VMKernel is shutting down

    Note:     The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

  • esxcli vsan cluster get shows vSAN is disabled

  • From vCenter > vSAN Cluster > Configure > vSAN  > Services vSAN shows as disabled

Environment

VMware vSAN (All Versions)

Cause

vSAN was disabled during the shutdown of the cluster.

Resolution

1) Navigate to vSAN Cluster > vSAN > Services and click on "Configure vSAN" for 7.x, for 8.x and newer select the cluster type to be configured.
Note: This will not reformat the existing disk groups/storage pools on the hosts. As you walk through the enable vSAN wizard the "Claim Disks" step will see the disks are already claimed and show the vSAN datastore information and you can just proceed to the next step in the wizard without issue.

2) Run the below commands on all hosts in the cluster
esxcfg-advcfg -d /VSAN/IgnoreClusterMemberListUpdates
 
3) Once vSAN is re-enabled vSAN > Services may show the "Shutdown Cluster" wizard is out of sync with the cluster
Note: For 8.x and newer if you get one of the below errors and vSAN fails to enable it means you chose the wrong vSAN storage architecture 
    • General vSAN error. vSAN ESA can not be enabled as diskgroups are present on 'esxi1,esxi2,esxi3,esxi4'. <--Re-enable vSAN choosing OSA as disk groups are present
    • General vSAN error. vSAN can not be enabled as there are storage pool disks on 'esxi1,esxi2,esxi3,esxi4'. <--Re-enable vSAN choosing ESA as storage pools are present
4) Once vSAN is re-enabled in vCenter run "esxcli vsan cluster get" to check if vSAN was enabled on the hosts.

5) Reboot vCenter to get vCenter and ESXi back in sync with each other
 
 
If further assistance is required open a case with vSAN support.

Additional Information

vSAN shows as disabled in vCenter or via ESXi CLI upon powering up the cluster after a planned cluster shutdown
Unable to restart a vSAN Cluster after manual shutdown using reboot_helper.py script

Powered by Wolken Software