Hosts cannot enter vSAN Maintenance mode- stuck in DECOM state 9.

Products

• VMware vSAN

Issue/Introduction

• "Host Maintenance Mode" warning in vSAN Health Check indicating that the vSAN Decom state is not in sync with Host Maintenance Mode. 
• vSAN objects maybe unhealthy and some storage related operations may fail- e.g. snapshot creation, changing storage policies, VM deployment.
• Host cannot enter or exit Maintenance mode from the UI or CLI
• Checking the host decomState returns a value of '9'.
• In some versions of vSphere you may see this extra information in the Health Check UI:
  
  • Host In Maintenance Mode: No
  • vSAN Node In Decommission State: In progress

You can check the host decomState from command line on the ESX host.

1. Get the host UUID by running cmmds-tool whoami.

2. Query the node's decommission state command:

   # cmmds-tool find -t NODE_DECOM_STATE -f json -u <host_uuid>

Environment

VCF 5.x

Cause

Decom state '9' means Maintenance Mode is in the process of cancelling. This status should not persist for more that a couple of seconds.  

Resolution

This status can be cleared by rebooting impacted hosts. 

Additional Information

A node is in vSAN Decommission State, even though the UI shows the ESXi host is not in Maintenance Mode

vSAN Health Service -- Host Maintenance Mode is in sync with vSAN Node Decommission State