Maintenance mode enabled vSAN ESXi host exits maintenance mode on its own post reboot

Products

• VMware vSAN

Issue/Introduction

Prior to this issue occurring a vSAN cluster shutdown and restart was performed.

Symptoms:

• vSAN ESXi host is placed in maintenance mode and rebooted, it exits maintenance mode on its own at boot.
• Only one host in the vSAN cluster experiences this issue.
• vSAN cluster has been restarted and is working normally.

Environment

VMware vSAN 8.x

Cause

• The host facing this issue was the Orchestrator host during the last vSAN cluster shutdown. And it currently still assumes itself to be the orchestrator host even after the vSAN cluster has been restarted successfully.
• This can be seen in the host's var/run/log/vsanmgmt.log:
  

  YYYY-MM-DDTHH:MM:SS.SSSZ In(14) vsand[2100559]: [opID=5601a389 VsanClusterPowerSystemImpl::PerformOrchestrationCluster
PowerAction] task 'vim.Task:vmodlTask-ha-host-34275964', spec (vim.cluster.PerformClusterPowerActionSpec) {
YYYY-MM-DDTHH:MM:SS.SSSZ In(14)[+] vsand[2100559]:   targetPowerStatus = 'clusterPoweredOn',
YYYY-MM-DDTHH:MM:SS.SSSZ In(14)[+] vsand[2100559]:   isOrchestration = true,
YYYY-MM-DDTHH:MM.SS.SSSZ In(14)[+] vsand[2100559]:   initialPowerStatus = 'hostsOutOfMM'
YYYY-MM-DDTHH:MM:SS.SSSZ In(14)[+] vsand[2100559]: }, vcVM 'vim.VirtualMachine:13'

• This is the normal workflow when the vSAN cluster is being restarted after a shutdown. The host on which the vCenter resides assumes the role of Orchestrator and orchestrates the vSAN cluster shutdown and restart process.
• The configuration entry which states the host is the orchestrator host is present in the /etc/vmware/vsan/vsanperf.conf file called "vc_vm_moid":
  

  hotspot_max_day_time_range = 7
long_retention_stats_mode_interval_in_minutes = 10080
vc_vm_moid = 13

• Due to this entry, the host assumes itself as orchestrator and performs the Cluster Power up sequence and exits host 1 out of maintenance mode.

Resolution

To resolve this issue:

1. Open SSH to the affected ESXI host.
2. Create a backup of the vsanperf.conf file: cp /etc/vmware/vsan/vsanperf.conf /etc/vmware/vsan/vsanperf.bkp
3. Remove the "vc_vm_moid" line from the /etc/vmware/vsan/vsanperf.conf file using vi editor.
4. Restart vsanmgmtd service - /etc/init.d/vsanmgmtd restart

Now on placing the host into maintenance mode and rebooting, it will continue to remain in maintenance mode.