vSAN 2-Node Cluster Shutdown Workflow Stuck at “Power Off All System VMs”

Products

• VMware vSAN

Issue/Introduction

Symptoms

• During a planned shutdown of a vSAN 2-node cluster, the shutdown process became stuck at the "Power off all system VMs" stage in the vSAN UI. The cluster was manually powered off without completing the full shutdown procedure via the vSAN interface.

Upon powering the cluster back on:

• Virtual machines were unable to vMotion.
• The advanced vSAN parameters DOMPauseAllCCPs and IgnoreClusterMemberListUpdates were both set to 0.
• All vSAN objects were in a healthy state.
• The cluster membership was correct, reflecting 3 members (2 hosts and 1 witness).
• An automated maintenance mode task remained active and stuck on the ESXi host(s).

Environment

VMware vSAN 7.x

Cause

The issue occurred because the vSAN cluster was powered off manually without completing the shutdown workflow via the vSAN UI. This left internal vSAN states incomplete, causing post power-on issues like vMotion failures.

Resolution

Workaround

• Confirm no shutdown-related tasks were stuck in the vCenter UI.
• All virtual machines were verified to be powered on and operational.
• Access the Managed Object Browser (MOB) and updated the cluster power status to "Powered On".
  • Access MOB:
    • https://<vCenter FQDN>/vsan/mob/?moid=vsan-duster-power-system&method=updateClusterPowerStatus
  • Copy only the domain-c<number> part of the URL.
    • https://<fqdn-of-vCenter-server>/ui/app/cluster;nav=h/urn:vmomi:ClusterComputeResource:domain-c####:ce4a7b9f-768c-2222-3333-############/summary.

• Following this update, the vSAN Services tab began displaying data correctly in the vSphere UI.

If the issue persists after performing the above steps, it is recommended to open a support case with Broadcom Technical Support for further assistance

Additional Information

• The issue is commonly seen when the shutdown process is manually interrupted or not fully completed via the vSAN UI.
• It is important to avoid manually powering off the cluster during the shutdown process to ensure all internal vSAN tasks and states are completed properly.