Maintenance Mode % progress stages in vSAN nodes.
search cancel

Maintenance Mode % progress stages in vSAN nodes.

book

Article ID: 326563

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

This article provides information related to processes taking place while a vSAN node is entering Maintenance Mode and troubleshooting directions related to each stage.

 

Symptoms:


vSAN-enabled ESXi hosts can take significant amount of time to complete (or fail to progress past) a specific stage of Maintenance Mode. 

Environment

VMware vSAN 7.0.x
VMware vSAN 8.0.x
VMware vSAN 9.0.x

Cause

Sending a vSAN-enabled ESXi host into Maintenance Mode has a number of implications and the processes that occur can vary depending on which Maintenance Mode option is taken (e.g. Full Data Evacuation, Ensure Accessibility or No Action).

Resolution

Progress percentages can help determine which task is currently underway (and potential blocker if progress appears halted):

0% - Task initializing

If the task is staying at this % for a long duration, this can indicate issues with vCenter or communication issues between vCenter and the hosts.


2% - Precheck.

Task can hang at this % if there are HA/DRS settings that might prevent the host from entering Maintenance Mode.


19% - vMotion of VMs off the host.

If the task is stuck at this % then user may need to intervene and manually migrate or power off (whichever is applicable) the remaining Virtual Machines. Not-progressing task at this stage does usually indicate one of the following conditions preventing VM migration: 

  • Passthrough device attached to a VM (e.g. GPU)
  • CD/DVD device pointing to ISO image no longer existing on the datastore
  • Virtual Machine network is not available on destination host(s)
  • VM disk(s) are stored on local-only datastore
  • Affinity/Anti-Affinity rules configuration preventing vMotion of a VM(s)
  • Insufficient compute resources on destination host(s) - either due to host overutilization or HA reservations settings 

20% - 100% - Resync and/or migration of data onto other nodes.

This stage can take a significant amount of time, depending on either

  • the amount of data that needs to be copied to other nodes
  • and/or the available storage resources for component placement

During this stage Cluster Monitoring, Membership, and Directory Service (CMMDS) will denote the host's 'decommissioning state' as '4' ("decomState": 4) and can be verified via CLI by running:

cmmds-tool find -f json -t NODE_DECOM_STATE -u $(cmmds-tool whoami) | grep content

...on the node in question.

Additional Information

In addition the vSAN maintenance mode progress can be monitored via ESXi's /var/run/log/clomd.log, user may see the messages similar to:

CLOM_ProcessDecomUpdate: Node 00000000-0000-0000-0000-000000000000 state change. Old:DECOM_STATE_NONE New:DECOM_STATE_ACTIVE Mode:1 JobUuid:00000000-0000-0000-0000-000000000000
CLOM_ProcessDecomUpdate: Node 00000000-0000-0000-0000-000000000000 state change. Old:DECOM_STATE_ACTIVE New:DECOM_STATE_INITIALIZED Mode:1 JobUuid:00000000-0000-0000-0000-000000000000
CLOM_ProcessDecomUpdate: Node 00000000-0000-0000-0000-000000000000 state change. Old:DECOM_STATE_INITIALIZED New:DECOM_STATE_DOM_READY Mode:1 JobUuid:00000000-0000-0000-0000-000000000000
CLOM_ProcessDecomUpdate: Node 00000000-0000-0000-0000-000000000000 state change. Old:DECOM_STATE_DOM_READY New:DECOM_STATE_PREP_COMPLETE Mode:1 JobUuid:00000000-0000-0000-0000-000000000000
CLOM_ProcessDecomUpdate: Node 00000000-0000-0000-0000-000000000000state change. Old:DECOM_STATE_PREP_COMPLETE New:DECOM_STATE_COMPLETE Mode:1 JobUuid:00000000-0000-0000-0000-000000000000

(DECOM_STATE_INITIALIZED denotes that the progress has started and DECOM_STATE_COMPLETE outlines successful completion of the task. )



 
Powered by Wolken Software