Virtual Machine operations fail on ESXi with error "The operation is not allowed in the current state" due to partial maintenance mode

Products

VMware vSphere ESXi

Issue/Introduction

Operations like Power on/off or Snapshot attempts on virtual machines residing on affected ESXi fail with the error message "The operation is not allowed in the current state" with hostd.log entries similar to below:

/var/run/log/hostd.log :

YYYY-MM-DDThh:mm:ss.msZ Wa(164) Hostd[2099474] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<datastore path>/<VM Path>/<VM Name>.vmx opID=######-##-#### sid=######## user=##] State Transition (VM_STATE_ON -> VM_STATE_CREATE
_SNAPSHOT) not allowed for this Vm : Blocked due to host (partial) maintenance mode state

YYYY-MM-DDThh:mm:ss.msZ Wa(164) Hostd[2099527]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/<datastore path>/<VM Path>/<VM Name>.vmx opID=######-##-#### sid=######## user=##] State Transition (VM_STATE_OFF -> VM_STATE_POWERING_ON) not allowed for this Vm : Blocked due to host (partial) maintenance mode state

vobd logs show entries of host entering Maintenance even though from vSphere or Host Client affected host does not seem to be in Maintenance mode

/var/run/log/vobd.log

YYYY-MM-DDThh:mm:ss.msZ In(14) vobd[2097957]: [GenericCorrelator] 1034996113090us: [vob.user.maintenancemode.entering] The host has begun entering maintenance mode

Maintenance mode command might have been issued from on affected ESXi host through cli that can be seen on shell log, causing host to enter into partial maintenance waiting for VMs to be evacuated by DRS

/var/run/log/shell.log

YYYY-MM-DDThh:mm:ss.msZ In(14) shell[5436667]: [root]: esxcli system maintenanceMode set --enable true

Environment

vSphere ESXi 8.0u3

Cause

The virtual machine operations fail as Host is stuck in Partial Maintenance mode which is a feature of ESXi 8.0u3.

In case Maintenance Mode command was issued from the Host CLI, virtual machines are not evacuated by DRS unless the command is issued from vCenter causing Host to be stuck in partial Maintenance mode.