ESXi task stalled at 82% while entering vSAN Maintenance Mode on vSphere Client

Products

• VMware vCenter Server
• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

• Entering Maintenance Mode task is stuck at 82% on vSphere client for ESXi host in vSAN Cluster
• Task details on the UI may report 

Objects Evacuated: <completed object count> of <total object count>, Data Evacuated: <object size> MB of <object size> MB

• /var/run/log/clomd.log: (on ESXi host)

YYYY-MM-DDTHH:MM:SS Er(27) clomd[PID]: [Originator@6876] CLOMDecomAffObjCb: Obj <Object ID> is not ready. configCSN: 439 policyCSN: 440 stateCSN: 439 configSCSN: 374 statsSCSN: 374, incompleteCmmdsState: 0.
YYYY-MM-DDTHH:MM:SS Er(27) clomd[PID]: [Originator@6876] CLOMDecomAffObjCb: Obj <Object ID> is not ready. configCSN: 439 policyCSN: 440 stateCSN: 440 configSCSN: 374 statsSCSN: 374, incompleteCmmdsState: 0.
YYYY-MM-DDTHH:MM:SS Er(27) clomd[PID]: [Originator@6876] CLOMDecomAffObjCb: Obj <Object ID> is not ready. configCSN: 286 policyCSN: 287 stateCSN: 286 configSCSN: 0 statsSCSN: 0, incompleteCmmdsState: 0.
YYYY-MM-DDTHH:MM:SS Er(27) clomd[PID]: [Originator@6876] CLOMDecomAffObjCb: Obj <Object ID> is not ready. configCSN: 286 policyCSN: 287 stateCSN: 287 configSCSN: 0 statsSCSN: 0, incompleteCmmdsState: 0.
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomMonitor: Decom monitor started.
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomPublishDecomStateEntryInt: Node <vSAN Node ID>: Publishing state DECOM_STATE_STARTED
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomPublishDecomStateEntryInt: Node <vSAN Node ID>: Published state DECOM_STATE_STARTED. Affected objects: 1, Progress: 71, Update: 1, numObjectsEvacuated 619, totalObjectstoEvacuate 620, bytesToSyncRemaining <bytes pending>, totalBytesToSync <total bytes of object>
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomMonitor: Decom monitor exited, rethunk: 1.
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomCMMDSResponseCb: CMMDS update response received: Success
YYYY-MM-DDTHH:MM:SS No(29) clomd[PID]: [Originator@6876] CLOMDecomCMMDSResponseCb: Node <vSAN Node ID>: CMMDS published decom state DECOM_STATE_STARTED Pending decom state DECOM_STATE_INVALID

Cause

The issue is caused due to a vSAN object sync taking longer than expected. The error DECOM_STATE_INVALID indicates a stuck state within the vSAN Control Plane.

Validate the same using the vCenter Server UI

vCenter UI > Cluster > Monitor > vSAN > Resyncing Object will show the amount of data to resync and estimated time to complete the resync.

Alternatively, validate using the below commands via ssh session to the ESXi host

localcli vsan debug resync summary get

ResyncSummary:
   Total Number Of Resyncing Objects: 1
   Total Bytes Left To Resync: <bytes left>
   Total GB Left To Resync: <object size in GB>

Resolution

In case of an active data transfer is identified, Do not restart services on the ESXi host.

Alternatively, reach out to Broadcom Support for further assistance.

Additional Information

vSAN Host fails to enter maintenance mode with ensure accessibility mode in a Stretched Cluster setup