vSAN objects unhealthy due to DECOM state

Products

• VMware vSAN

Issue/Introduction

Impact/Risks:
 
Any host stuck in DECOM STATE: 6 will not be contributing storage or compute resources to the cluster because vSAN believes the host to still be in Maintenance Mode.

 

Symptoms:

• Virtual Machines become invalid, inaccessible or orphaned in vSphere.
  
  
• The vSAN objects may be shown as reduced availability.
  
  
• May see error as "There are currently X usable fault domains. The operation requires Y more usable fault domains" while creating VMs/ snapshots.
  
  
• All the drives in hosts are mounted in cmmds, and none of the hosts are network partitioned.
  
  
• vSAN objects maybe unhealthy and some storage related operations may fail- e.g. snapshot creation, changing storage policies, VM deployment.
  
  
• If hosts are in a vSAN decom state, the available space on the vSAN datastore may show diminished.
  
  
• Unable to bring up vCenter or any other VM. Might see the error "Operation is not allowed in the current state" when attempting to power on.
  
  
• From the command "localcli vsan cluster get" will see Maintenance Mode Enabled or "ON" as below:
  
    Cluster Information:
    Enabled: true
    Current Local Time: 2018-09-14T18:51:57Z
    Local Node UUID: ########-####-####-####-########a270
    Local Node Type: NORMAL
    Local Node State: AGENT
    Local Node Health State: HEALTHY
    Sub-Cluster Master UUID: ########-####-####-####-########a4dcool ... 0
    Sub-Cluster Backup UUID: ########-####-####-####-########a390
    Sub-Cluster UUID: ########-####-####-####-########f8c7
    Sub-Cluster Membership Entry Revision: 3
    Sub-Cluster Member Count: 4
    Sub-Cluster Member UUIDs: ########-####-####-####-########a390, ########-####-####-####-########a4d0, ########-####-####-####-########c7b0, ########-####-####-####-########a270
    Sub-Cluster Membership UUID: ########-####-####-####-########a4d0
    Unicast Mode Enabled: true
    Maintenance Mode State: ON  <<---This node is in Decom state according to vSAN.
    Config Generation: ########-####-####-####-########112e 5 2018-09-14T17:21:34.629
  
  
• Use the following script to see which Host(s) in cluster are in DECOM state (indicated by a value of "decomState": 6)
  
  
  • echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

hostname,decomState,decomJobType
esxi1,6,0  >> DECOM STATE 6
esxi2,0,0
esxi3,0,0
esxi4,0,0
esxi5,0,0
esxi6,0,0
    
     For more information about the various decommission states 
     

    Decommission State	Meaning
    0	None - the node is not decommissioned
    1	The decommissioning process has been started
    3	The decommissioning process is underway
    6	The node has been decommissioned

• Referencing vSAN Skyline Health you see the alert "Host Maintenance Mode"

 

Environment

VMware vSAN 7.x
VMware vSAN 8.X

Cause

• This issue is frequently the result of issuing a maintenance mode task in vCenter, quickly followed by a cancellation of the maintenance mode task.
• This issue can also occur if the vSAN cluster was not shutdown properly before a planned shutdown/maintenance activity.
• The host may end up stuck in vSAN Decom State: 6 where vSAN considers the host to still be in Maintenance Mode.

Resolution

To clear the DECOM state of the ESXi host(s), follow the below steps:
 
1) Place the affected host into maintenance mode with No Action or No Data Migration.
2) Remove the host from maintenance mode.
3) Verify Object health.
4) Sometimes the host will not exit from DECOM_STATE after reboot.

In this situation shutdown the host completely ( check the dependency by debug precheck command beforehand ex. esxcli vsan debug evacuation precheck -e " HOSTNAME/UUID") . Power off the host wait for 20 seconds and Power On again.

Additional Information

vSAN and vSphere maintenance modes may diverge