vSAN Host Stuck Entering Maintenance Mode Due to Orphaned Canceled Decommission State - VMware vSAN

Products

• VMware vSAN

Issue/Introduction

A vSAN cluster may fail to enter maintenance mode, with the task stalling at 0% or 87%, and showing "Objects Evacuated: 0 of 0" in vCenter, even after all virtual machines have migrated.
This issue is typically caused by a stale or "orphaned" canceled decommission state on a participating host, which prevents the cluster from completing the necessary pre-check operations for maintenance mode.

Environment

• VMware vSAN (All versions, specifically 8.x ESA/OSA)
• Affected hosts stuck in an inconsistent decommission state.

Cause

A host within the cluster has a stale "canceled" vSAN maintenance mode state in the CMMDS (Cluster Monitoring, Membership, and Directory Service). Because the cluster requires all nodes to report a clean state before proceeding with a cluster-wide maintenance mode operation, this orphaned state blocks the maintenance mode request on all hosts in the cluster.

To confirm the host is stuck in the canceled decommission state, check the /var/log/vsansystem.log for the following pattern:

 Wa(164) vsansystem[2100218]: [vSAN@6876 sub=VsanSystemProvider opId=##########] Canceling in-progress node decommissioning operation
 In(166) vsansystem[2100218]: [vSAN@6876 sub=VsanSystemProvider opId=##########] Current decom state: 9
 In(166) vsansystem[2100218]: [vSAN@6876 sub=VsanSystemProvider opId=##########] Decom state: 9, waiting for cleanup state change to failed

Resolution

1. Identify the affected host: Run the following command from an ESXi host CLI to check the decommission state of all hosts in the cluster. Look for any node reporting decomState as 9:

   echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

    

   Example output:

   hostname,decomState,decomJobType
   ######## ,0,0
   ######## ,0,0
   ######## ,9,0
   ######## ,0,0
   ######## ,0,0
   ######## ,0,0
   

    

   Restart the clomd service: Log in to the ESXi host identified as having the stale state and run the following command: /etc/init.d/clomd restart

2. Clear pending tasks: If the maintenance mode task is still active on another host, restart the clomd service on that host as well.

3. Verify cluster state: Re-run the command in Step 1 to ensure all hosts now report decomState as 0.

4. Restore Monitoring: Re-enable any silenced vSAN health check alarms in the environment to ensure proactive notification if this condition recurs.

5. Complete Maintenance Mode: Attempt to place the host into maintenance mode again through vCenter.

Additional Information

See KB vSAN and vSphere maintenance modes may diverge for additional information