ESXi is stuck in maintenance mode at 32% without any Progress due to Stale/Orphaned vCLS VM

Products

• VMware vCenter Server 8.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• The ESXi host becomes stuck at 32% while entering maintenance mode, displaying a status "Waiting for all VMs to be powered off or suspended or migrated. In a DRS cluster check the Faults page on the DRS tab for troubleshooting."
• Reconfiguration of  vSphere cluster service status from Retreat mode and changing it to System Managed mode still has Orphaned vCLS VM in Unknown State.

         

Environment

VMware ESXi 8.0.3

VMware vCenter Server 8.0.3

Cause

A corrupted or orphaned vCLS virtual machine is preventing vSphere Cluster Services from functioning normally. Because vCLS is required for vSphere Distributed Resource Scheduler (DRS) operation, DRS cannot automatically evacuate the virtual machines from the host, causing the maintenance mode task to stall indefinitely.

Resolution

Remove the orphaned vCLS virtual machine to allow the vCenter Server to automatically redeploy a healthy vCLS instance.

1. SSH to the vCenter server as root.

2. Restart the vCenter vpxd service : service-control --stop vmware-vpxd && service-control --start vmware-vpxd.

3. Once the stale vCLS registration is cleared, vCenter will automatically deploy a new, healthy vCLS VM.

4. Verify vCLS health returns to a healthy state under Cluster > Monitor > vSphere Cluster Services (vCLS) > Health.

5. Retry placing the ESXi host into maintenance mode.

Additional Information

Reference KB's:

ESXi host stuck "in progress" when entering maintenance mode

Receiving error about DRS being in a failed state - Error: vSphere DRS functionality was impacted due to unhealthy state of vSphere cluster services

Manually Destroy Embedded vCLS on vSphere 8.0 U3 and 9.x ESXi Hosts

vSphere Cluster Services (vCLS) Known Issues/Corner Cases