ESXi host stuck "in progress" when entering maintenance mode

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• When attempting to place an ESXi host into maintenance mode via the vSphere Client, the task hangs indefinitely in the "In Progress" state.
  
  
  • Virtual machines are not migrating off the host.
    
    
  • The host is unresponsive to any other vCenter tasks.
    
    
  • The hung maintenance mode task often cannot be canceled.
    
    
• The task progress bar is stalled, displaying the following message: "Waiting for all virtual machines to be powered off or suspended or migrated. In a DRS cluster check the Faults page on the DRS tab for troubleshooting"

Environment

VMware vCenter Server 8.0
VMware vSphere ESXi 8.0

Cause

A host fails to enter maintenance mode because resident virtual machines are unable to power off or migrate successfully. This typically occurs due to one or more of the following reasons:

• Distributed Resource Scheduler (DRS) is either disabled or not set to Fully Automated mode, preventing the system from automatically migrating virtual machines off the host.
  
  
• Virtual machines have active configurations tying them to the current host, such as connected CD-ROM drives or dependencies on local storage.
  
  
• General vMotion errors are preventing the migration process. (Note: Performing a quick manual vMotion test on the affected virtual machines will often reveal the specific blocker).
  
  
• The AD user  logged into the vSphere Client was not part of the Administrator group. 

Resolution

If the ESXi host remains indefinitely stuck at "In Progress" without timing out, yet appears to be operating normally when accessed directly via the host client, take the following steps:

1. Check the cluster's DRS configuration. If DRS is disabled or not set to fully automated, cancel the pending maintenance mode task, update the settings to fully automated, and try again.
   
   
2. Refresh the vSphere Client page if the maintenance mode action is not visible or if a cancellation attempt fails to process.
   
   
3. Check the virtual machines tab on the host to determine if only a partial migration occurred. For any remaining virtual machines, initiate a manual migration and monitor the pre-check validation for compatibility errors or other conditions blocking the move.
   
   
4. Verify that the date and time configurations are accurate and uniform across the vCenter Server and all associated ESXi hosts. Time synchronization discrepancies frequently result in a loss of communication between the host and vCenter.
   
   
   1. vCenter: Access the VAMI interface (https://<vcenter-ip-or-fqdn>:5480) and review the Date/Time configuration.
   2. ESXi: Navigate to the Configuration tab for the specific host.
      
      
5. If the issue persists after completing the steps above, restart the management services of the ESXi by following the instructions in the article: Restarting the Management agents in ESXi
   
   
6. If a stale task remains stuck in vCenter, restart the vsphere-ui and vmware-vpxd services by following the instructions in this KB article : Clear stale tasks from the vCenter "Recent Tasks" panel
   Important: Take a snapshot of the vCenter Server virtual machine before restarting the services.
   
   

7. If the user is not part of the Administrator group then add user as a part of Administrator group to have access on vCLS VM and cancel and re-initiate the maintenance mode task. Follow the steps below to add the required user to the Administrator group:

   • Log in to the vSphere Client and navigate to Menu.

   • Select Administration under the Menu options.

   • In the Single Sign-On section, click on Users and Groups and then on Groups.

   • Select the Administrator group and click Edit.

   • Add the user who needs to manage and monitor the vCLS VMs.

Additional Information

ESXi is stuck in maintenance mode at 32% without any Progress due to Stale/Orphaned vCLS VM