ESXi Host exits maintenance mode unexpectedly after reboot via IPMI

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• An ESXi host was placed into maintenance mode and subsequently rebooted via IPMI. Post-reboot, the host was observed to be no longer in maintenance mode, although no action was taken by administrators to exit maintenance mode.
• This unexpected behavior led to a discrepancy between the expected host state and what was reported in vCenter, raising concerns about configuration persistence and proper host management during reboots.
• The /var/log/vmware/vpxd/vpxd.log in vCenter will show the automatic exitMaintenanceMode task, even though it was not manually initiated.
  YYYY-MM-DDTHH:MM:SS info vpxd[#####] [Originator@6876 sub=vpxLro opID=HB-host-#####@###-########-##] [VpxLRO] -- BEGIN task-605342 -- example.host.com -- HostSystem.exitMaintenanceMode --
YYYY-MM-DDTHH:MM:SS info vpxd[#####] [Originator@6876 sub=MoHost opID=HB-host-#####@###-########-##] Ignore the WCP exitMaintenanceMode API result for host [vim.HostSystem:host-#####, hostname] because vSphere Namespaces is not enabled on its parent cluster.

• The ESXi /var/run/log/vobd.log will not show a corresponding exit event, confirming the state was not explicitly changed on the host itself.
• hostd.log under the location/var/run/log in ESXi shows the enter maintenance task is executed:

YYYY-MM-DDTHH:MM:SS info hostd[2100356] [Originator@6876 sub=Hostsvc.ModeMgr opID=me6k####-380##1-auto-85po-h5:700###95-96-##-3##8 user=vpxuser:username] Enter: next = maintenance, current = normal, count = 0, timeout = 0

YYYY-MM-DDTHH:MM:SS warning hostd[2101176] [Originator@6876 sub=Hostsvc.ModeMgr opID=me6k####-380##1-auto-85po-h5:700###95-96-##-3##8] Transition 'haTask-ha-host-vim.HostSystem.enterMaintenanceMode-8423310' already finished, current is ''

YYYY-MM-DDTHH:MM:SS info hostd[3491633] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
-->    key = 144,
-->    chainId = 1173179864,
-->    createdTime = "1970-01-01T00:00:00Z",
-->    userName = "",
-->    host = (vim.event.HostEventArgument) {
-->       name = "hostname",
-->       host = 'vim.HostSystem:ha-host'
-->    },
-->    eventTypeId = "esx.audit.maintenancemode.entering",
-->    objectId = "ha-host",
-->    objectType = "vim.HostSystem",
--> }
YYYY-MM-DDTHH:MM:SS info hostd[3491633] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 26449 : The host has begun entering maintenance mode.
YYYY-MM-DDTHH:MM:SS info hostd[2100356] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=me6k####-380##1-auto-85po-h5:700###95-96-##-3##8 user=vpxuser:username] Event 26450 : Host hostname in ha-datacenter has started to enter maintenance mode

Environment

• VMware vCenter Server 8.0.x
• VMware vCenter Server 7.0.x
• VMware vSphere ESXi 8.0.x
• VMware vSphere ESXi 7.0.x

Cause

• The issue occurs because the reboot was initiated via IPMI, bypassing the ESXi OS-level shutdown process.
• As a result, the backup.sh script, which is responsible for persisting runtime configuration (including maintenance mode status), did not run.
• This caused the ESXi host to lose its maintenance mode state after the reboot. Once the host came back online and reported its state to vCenter, vCenter detected that the host was not in maintenance mode and triggered an automatic exitMaintenanceMode task to match the host’s reported state.

Resolution

This issue is not a bug, but a known behavior when using non-graceful shutdown/reboot methods for ESXi host.

• Always initiate reboots using one of the following methods:
  • vCenter UI
  • ESXi Host Client UI
  • Direct Console User Interface (DCUI)

Note : These methods ensure that the backup.sh script runs, preserving runtime configuration in state.tgz (bootbank), including the maintenance mode status.

• Use IPMI reboot only as a last resort, when:
  • The host is unresponsive to vCenter.
  • The host UI and SSH access are unavailable.

• Verify host state post-reboot when using IPMI:
  • Via vCenter or ESXi Host Client.
  • From ESXi host SSH session :
    • less /etc/vmware/hostd/hostsvc.xml | grep -i maintenance
      • Output : <mode>maintenance</mode>
    • esxcli system maintenanceMode get
      • Output : Enabled

• By following these best practices, administrators can ensure that maintenance mode is preserved during reboots and avoid unintended exposure of ESXi hosts to workloads.