ESXi hosts get stuck at 19% during entering maintenance mode due to hanging vCLS control plane dependencies.
search cancel

ESXi hosts get stuck at 19% during entering maintenance mode due to hanging vCLS control plane dependencies.

book

Article ID: 399791

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESXi 8.0 VMware vSphere ESX 8.x

Issue/Introduction

  • When attempting to place two ESXi hosts within a cluster into maintenance mode, the operation remained stuck at 19%. The vSphere Client displays a message indicating a dependency on a VM that had already been migrated:
    "Waiting for <VM_Name> to complete task: Drm.ExecuteVMotionLRO"

  • Additionally, in the task history, the following message will be observed:
    "The user does not have permission to view the entity associated with this event."

  • Although the VM referenced in the error message had already been evacuated, internal logs showed that the maintenance mode operation neither completed nor failed with an explicit error. The logs also indicated that vCLS was attempting to disable its components on the affected host, which likely blocked the maintenance mode operation.

  • Relevant log excerpts on vCenter, /var/log/vmware/vpxd/vpxd.log
    info vpxd [...] Ignore the WCP enterMaintenanceMode API result for host [...] because vSphere Namespaces is not enabled on its parent cluster.
    info vpxd [...] Started retries to disable vCLS Pod Crx on host [...]
    ..
    ..
    Host has powered-on system VMs; [vim.HostSystem:host-[...],[...]], eam: false, vclscrx: true

Environment

VMware vCenter Server 8.x

Cause

Host remained in an intermediate state while the system attempted to handle vCLS dependencies, which stalled the maintenance mode transition.

The root cause is configuration drift within vCenter's configuration file (/etc/vmware-vpx/vpxd.cfg). Stale XML blocks under the <vcls> tag—retained from previous software versions or failed state transitions—create a state-machine mismatch between vCenter (vpxd) and the ESXi host container runtime. Because of this structural conflict, vCenter cannot successfully issue or complete the teardown signal for the Embedded vCLS CRX pods during host maintenance entry.

Upgrading to vSphere 8.0 Update 3 frequently leaves residual legacy configuration tags in vpxd.cfg. When this happens, the workaround silently fails because the vCenter state machine gets trapped in a loop.

Resolution

Workaround 1:

  1. Log in to the vCenter UI with an Administrator user.
  2. Identify the vCLS VM on the host that is stuck at 19% entering maintenance mode.
  3. Place Cluster vCLS into Retreat Mode
  4. Power off the vCLS VM if they are not automatically cleaned up.
  5. Host will now complete entering Maintenance Mode.
  6. Reboot the ESXi Host.
  7. Exit Maintenance Mode.
  8. Ensure vSAN Skyline Health is 100%
  9. Place Cluster vCLS back into System Managed Mode.

Refer to this doc if the Host is part of a VSAN Node, to place it in Maintenance mode - Place a Member of vSAN Cluster in Maintenance Mode

Workaround 2:

  1. Take a snapshot of the vCenter Server Appliance prior to modifying core system files.

  2. Sanitize vpxd.cfg Manually:

    • Stop the vCenter Server daemon:

      service-control --stop vpxd

    • Backup the configuration file:

      cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak

    • Open /etc/vmware-vpx/vpxd.cfg and safely remove the stale <vcls>...</vcls> XML block. (Note: Avoid greedy line-deletion commands such as multiline sed to prevent stripping unintended configuration tags).

    • Start the vCenter Server daemon:

      service-control --start vpxd

  3. Execute Retreat Mode Cycle:

    • Navigate to vSphere Client > [Cluster] > Configure > vSphere Cluster Services > General.

    • Click EDIT VCLS MODE, switch to Retreat Mode, and save. Verify that the Embedded vCLS CRX pods are completely removed from inventory.

  4. Complete Host Maintenance: Place the target ESXi host into Maintenance Mode (which now completes smoothly past 19%), execute required updates, and exit Maintenance Mode.

  5. Re-Enable System Managed Mode: Return to vSphere Cluster Services > General, click EDIT VCLS MODE, and set it back to System Managed to deploy fresh Embedded vCLS pods.

Additional Information

  • The vCLS VM cannot be manually migrated or powered off via standard methods. It is managed automatically by the vSphere control plane.

  • Post-reboot, maintenance mode operations completed successfully on the affected hosts.

  • Logs did not provide a definitive root cause, but symptoms point toward a vCLS-related blockage.

Powered by Wolken Software