ESXi host stuck "in progress" when entering maintenance mode

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• When attempting to place an ESXi host into maintenance mode via the vSphere Client, the task hangs indefinitely in the "In Progress" state.
  • Virtual machines are not migrating off the host.
  • The host is unresponsive to any other vCenter tasks.
  • The hung maintenance mode task often cannot be canceled.
• The task progress bar is stalled, displaying the following message:
  • "Waiting for all virtual machines to be powered off or suspended or migrated. In a DRS cluster check the Faults page on the DRS tab for troubleshooting"

Cause

A host may fail to enter maintenance mode because resident virtual machines are unable to power off or migrate successfully. This typically occurs due to one or more of the following reasons:

• Distributed Resource Scheduler (DRS) is either disabled or not set to Fully Automated mode, preventing the system from automatically migrating virtual machines off the host.
• Virtual machines have active configurations tying them to the current host, such as connected CD-ROM drives or dependencies on local storage.
• General vMotion errors are preventing the migration process. (Note: Performing a quick manual vMotion test on the affected virtual machines will often reveal the specific blocker).

Resolution

If the ESXi host remains indefinitely stuck at "In Progress" without timing out, yet appears to be operating normally when accessed directly via the host client, take the following steps:

• Check the cluster's DRS configuration. If DRS is disabled or not set to fully automated, cancel the pending maintenance mode task, update the settings to fully automated, and try again.
• Refresh the vSphere Client page if the maintenance mode action is not visible or if a cancellation attempt fails to process.
• Check the virtual machines tab on the host to determine if only a partial migration occurred. For any remaining virtual machines, initiate a manual migration and monitor the pre-check validation for compatibility errors or other conditions blocking the move.
• Verify that the date and time configurations are accurate and uniform across the vCenter Server and all associated ESXi hosts. Time synchronization discrepancies frequently result in a loss of communication between the host and vCenter.
  
  • vCenter: Access the VAMI interface (https://<vcenter-ip-or-fqdn>:5480) and review the Date/Time configuration.
  • ESXi: Navigate to the Configuration tab for the specific host.
• If the issue persists after completing the steps above, restart the management services of the ESXi by following the instructions in the article: Restarting the Management agents in ESXi
• If a stale task remains stuck in vCenter, restart the vsphere-ui and vmware-vpxd services by following the instructions in this KB article : Clear stale tasks from the vCenter "Recent Tasks" panel
  • Important: Take a snapshot of the vCenter Server virtual machine before restarting the services