Exiting an ESXi Host Stuck in Maintenance Mode

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Exit maintenance mode task fails in vCenter
• Exit maintenance mode task hangs and does not time-out or fail.
• The "Exit Maintenance Mode" option is unavailable or grayed out in the vSphere Client.

Environment

VMware vSphere ESXi 7.x / 8.x / 9.x

Cause

This behavior is typically caused by a synchronization failure between the ESXi host management daemons or stale/hung tasks remaining in the host's task manager.

Resolution

Use the following methods to exit a non-vSAN ESXi host from maintenance mode:

1. Access the ESXi host shell and run the following command to validate the maintenance mode status:
   vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode
   Reference: Using ESXi Shell in ESXi
2. Log directly into the ESXi host web UI as root and attempt exiting maintenance mode.
3. Restart the vSphere Web Client.
   Reference: Stopping, Starting or Restarting appliance services
4. Run the following command via SSH:
   vim-cmd /hostsvc/maintenance_mode_exit
   
5. Restart the host management agents and re-attempt exiting maintenance mode once the services have restarted. 
   Reference: Restarting the Management agents on an ESX or ESXi Server.

If the ESXI host is part of vSAN Cluster and vSAN health check reports the host is in maintenance mode, it is recommend to engage Broadcom Technical Support.
Reference: Contact Broadcom Support

Use the following methods to exit a vSAN host from maintenance mode:

1. Log directly into the ESXi host web UI as root and attempt exiting maintenance mode.
2. Run the following command through SSH to determine the hosts maintenance mode state:
   vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode
   esxcli vsan cluster get
3. If the host MaintenanceMode state is "ON", run the following command to exit maintenance mode:
   esxcli system maintenanceMode set -e false
4. If the host MaintenanceMode state is "ENTERING", run the following command to cancel the maintenance mode process:
   esxcli vsan maintenancemode cancel
5. From the Ruby vSphere Console (RVC), navigate to the host level and run:
   vsan.host_exit_evacuation
   Reference: Using Ruby vSphere Console (RVC) for vSAN operations/troubleshooting
6. If the methods above do not remove the host from maintenance mode, vSAN management agents can be restarted using the following steps: 
   1. Ensure the ESXi host is evacuated of all VMs
   2. Restart the host management services with the following commands through SSH:
      
      1. /etc/init.d/vpxa restart
      2. /etc/init.d/hostd restart
      3. /etc/init.d/vsanmgmtd restart
         • WARNING: Do not use the services.sh restart command or Restart Management agents from the Direct Console User Interface (DCUI) for vSAN hosts!
   3. After restarting services, re-attempt the methods above to exit maintenance mode
7. In rare conditions, the esxcli vsan cluster get command may return MaintenanceMode: OFF, but the ESXI host may be in matenance mode at the vSAN level. 
   1. To verify if host is in maintenance mode at vSAN, please run command:
      cmmds-tool find -t NODE_DECOM_STATE -f json
   2. Make sure ESXI host UUID and Owner ID match (example below)
      

        "entries":
      [
       {
         "uuid": "########-####-####-####-############",
         "owner": "########-####-####-####-############",
         "health": "healthy",
         "revision": "6",
         "type": "NODE_DECOM_STATE",
         "flag": "0",
         "md5sum": "941d5a542cf3e5f5b6f821f17b8c13c2",
         "valueLen": "64",
         "content": {"nodeDecomState": 6, "decomJobType": 0, "decomJobUuid": "########-####-####-####-############", "progress": 0, "affObjList": [ ], "errorCode": 0},
         "errorStr": "(null)"
       }

   3. If the nodeDecomState or decomJobType is any non-zero value, see the following KB to resolve the status divergence condition: vSAN and vSphere maintenance modes may diverge