vSAN Decommission State and ESXi Maintenance Mode Mismatch.

Products

• VMware vSAN

Issue/Introduction

• Cluster > Monitor > vSAN > Skyline Health reports alarms for "vSAN Object Health" and "Host Maintenance Mode" as below:
  
  

• Objects may stuck in a "Reduced Availability with No Rebuild" state.
  
  [root@ esx1:~]esxcli vsan debug object health summary get
Health Status                                              Number Of Objects
---------------------------------------------------------  -----------------
remoteAccessible                                                           0
inaccessible                                                               0
reduced-availability-with-no-rebuild                                      31
reduced-availability-with-no-rebuild-delay-timer                           0
reducedavailabilitywithpolicypending                                       0
reducedavailabilitywithpolicypendingfailed                                 0
reduced-availability-with-active-rebuild                                   0
reducedavailabilitywithpausedrebuild                                       0
data-move                                                                  0
nonavailability-related-reconfig                                           0
nonavailabilityrelatedincompliancewithpolicypending                        0
nonavailabilityrelatedincompliancewithpolicypendingfailed                  0
nonavailability-related-incompliance                                       0
nonavailabilityrelatedincompliancewithpausedrebuild                        0
healthy                                                                    0

• Running esxcli vsan cluster get on the affected host shows "Maintenance Mode State: ON"
  
  [root@ esx1:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 20##-0#-1#T1#:1#:39Z
   Local Node UUID: #######-####-####-####-##########
   Local Node Type: NORMAL
   Local Node State: AGENT
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: #######-####-####-####-##########
   Sub-Cluster Backup UUID: #######-####-####-####-##########
   Sub-Cluster UUID: #######-####-####-####-##########
   Sub-Cluster Membership Entry Revision: 1
   Sub-Cluster Member Count: 2
   Sub-Cluster Member UUIDs: #######-####-####-####-##########, #######-####-####-####-##########
   Sub-Cluster Member HostNames: esx1.example.local, esx2.example.local
   Unicast Mode Enabled: true
   Maintenance Mode State: ON
   Config Generation: #######-####-####-####-########## 1#6 20##-0#-1#T0#:4#:3#.239
   Mode: REGULAR
   vSAN ESA Enabled: false

• While esxcli system maintenanceMode get shows "Disabled" as below:

             [root@ esx1:~] esxcli system maintenanceMode get
              Disabled

Environment

VMware vSAN (All Versions)

Cause

A state mismatch occurs when the vSAN decommission metadata does not synchronize with the ESXi host's actual maintenance state. And is triggered by:

Rapidly initiating and canceling an ESXi maintenance mode task.
Management agent timeouts or network interruptions during data evacuation.
The host stuck in vSAN Decom State: 6. 

Output 
echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort
hostname,  decomState,  decomJobType
host-21,0,0
host-22,0,0
host-23,6,0
host-24,0,0
host-25,0,0

decomState 6 means the host is in vSAN Decom

Resolution

To re-synchronize the vSAN decommission status with the ESXi maintenance mode state:

1. Place the host into Maintenance Mode using the No Data Migration option.

2. Once the task completes, exit Maintenance Mode.

3. Verify the cluster state using command esxcli vsan cluster get (Confirm Maintenance Mode State is OFF)

4. Check object health using esxcli vsan debug object health summary get

Additional Information

vSAN Health Service -- Host Maintenance Mode is in sync with vSAN Node Decommission State