vSAN Cluster powered-on failed at 'Restart all system VMs in this cluster' phase

Products

• VMware vSAN

Issue/Introduction

vSAN Cluster powered-on failed at 'Restart all system VMs in this cluster' phase. 

vSphere Client shows error like this.

(vmodl.fault.ManagedObjectNotFound) { msg = "Received SOAP response fault from [<<io_obj p:0x00007f441ce46d28, h:83, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-###/vsan>]: updateClusterPowerStatus\n", faultMessage = (vmodl.LocalizableMessage) [], obj = 'vim.cluster.VsanClusterPowerSystem:ha-vsan-power-system' }

 

You could find supportedVMsPoweredOn failure from vmware-vsan-health-service.log.

YYYY-MM-DDThh:mm:ss.###+TZ INFO vsan-mgmt[18228] [VsanVcClusterPowerSystemImpl::_DoClusterPowerAction opID=agw-0000023-79c5] toAchievePowerStatus supportedVMsPoweredOn
YYYY-MM-DDThh:mm:ss.###+TZ ERROR vsan-mgmt[18228] [VsanVcClusterPowerSystemImpl::_DoClusterPowerAction opID=agw-0000023-79c5] _DoClusterPowerAction error: toAchievePowerStatus: supportedVMsPoweredOn
Traceback (most recent call last):
  File "bora/vsan/health/vpxd/pyMoVsan/VsanVcClusterPowerSystemImpl.py", line 544, in _DoClusterPowerAction
  File "bora/vsan/health/vpxd/pyMoVsan/VsanVcClusterPowerSystemImpl.py", line 174, in _UpdateSupportedVMs
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 598, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
PyCppVmomi.vmodl.fault.ManagedObjectNotFound: (vmodl.fault.ManagedObjectNotFound) {
  msg = "Received SOAP response fault from [<<io_obj p:0x00007fcc906c0f98, h:77, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vsan>]: updateClusterPowerStatus\n",
  faultMessage = (vmodl.LocalizableMessage) [],
  obj = 'vim.cluster.VsanClusterPowerSystem:ha-vsan-power-system'
}

Environment

VMware vSAN

Cause

This is a known issue fixed in ESXi 8.0 U1c or later.

After shutting down the cluster using the vSAN cluster wizard in the past, some hosts are experiencing issues with vsanmgmtd not running after another cluster shutdown. On those hosts, there are duplicate entries in /etc/vmware/vsan/vsanperf.conf, where the keys of those duplicate entries only differ in capitalization. The keys affected by this issue could be 'vc_vm_moId', 'allowSelfContain', and 'infra_vm_moIds'.

Resolution

Remove all the duplicate 'vc_vm_moId', 'vc_vm_moid', 'allowSelfContain", "allowselfcontain", "infra_vm_moIds" and "infra_vm_moids" entries in vsanperf.conf and restart vsanmgmtd. Then, restart cluster restart again.

Additional Information

vsanmgmtd stops running after shutting down the vSAN cluster via vCenter vSAN Cluster Shutdown Wizard