ESXi host partitioned from vSAN cluster due to empty unicastagent list and stale CMMDS

Products

• VMware vSAN

Issue/Introduction

• Skyline health may report with vSAN Cluster partition alarm under vSAN cluster>Monitor>Skyline health.

• An ESXi host becomes partitioned and is isolated from the vSAN cluster.
• The command "esxcli vsan cluster get" reports the sub-cluster membership count as 1, indicating the host is completely isolated from the rest of the cluster nodes. This command capture the Sub-Cluster member count that includes all the nodes running in the vSAN cluster.

[root@server name:~] esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTHH:MM:SS
   Local Node UUID: ########-####-####-####-############
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: ########-####-####-####-############
   Sub-Cluster Backup UUID: ########-####-####-####-############
   Sub-Cluster UUID: ########-####-####-####-############
   Sub-Cluster Membership Entry Revision: 37
   Sub-Cluster Member Count: 1  
   Sub-Cluster Member UUIDs: ########-####-####-####-############, ########-####-####-####-############
   Sub-Cluster Member HostNames: server name1, server name2
   Sub-Cluster Membership UUID: ########-####-####-####-############
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: ########-####-####-####-############ 7 YYYY-MM-DDTHH:MM:SS
   Mode: REGULAR

• Execution of "esxcli vsan cluster unicastagent list" on the affected host returns an empty output.

• After manually adding the unicastagent list entry, the Sub-Cluster Member Count can remain at 1.

 

Environment

VMware vSAN 8.x

VMware vSAN 9.x

Cause

The ESXi host partitioned due to an unpopulated vSAN unicastagent list, preventing peer communication. Subsequent cluster instability is driven by a stale or unresponsive CMMDS master node state.

Resolution

To resolve the vSAN cluster partition issue, the affected host must be properly rejoined to the vSAN cluster with the correct Sub-Cluster UUID.

• Force the partitioned ESXi host to leave the vSAN cluster by executing the following command on the affected node. This command acts as a hard reset for the host's vSAN cluster membership. 

esxcli vsan cluster leave

• Rejoin the affected host to the cluster using the appropriate sub-cluster UUID. It instructs the host's vSAN storage daemon to start listening to and participating in the specific communication domain shared by the healthy peer nodes.

esxcli vsan cluster join -u <sub-cluster uuid>

Note: Note down the sub-cluster UUID from peer node of the same vSAN cluster by running the command "esxcli vsan cluster get".

• Verify the unicastagent list whether it repopulates correctly by executing the below command.

esxcli vsan cluster unicastagent list

• If vSAN cluster instability persists even after the unicast agent list has successfully repopulated, it is likely that the affected node failed to synchronize its updated state with the active CMMDS list. To refresh the CMMDS directory and clear any stale cluster states, place the active CMMDS master node into Maintenance Mode.
  
  
• Force a CMMDS leader re-election to clear the stale state by executing the following command.

vsish: vsish -e set /vmkModules/cmmds/forceTransition abdicateLeader

 

Note:
Even if the network allows communication (unicast agents are populated) and the host knows the correct UUID, the cluster might still reject the host. This happens at the CMMDS layer.

One host in the cluster is always elected as the CMMDS Master. Sometimes, the Master node caches a "stale" state, incorrectly believing the partitioned host is still dead or incompatible, and refuses to let it sync. Executing the abdicateLeader command via vsish forces the Master node to immediately resign from its leadership role.

When the Master steps down, the cluster instantly holds a new election. The newly elected Master is forced to rebuild the CMMDS directory entirely from scratch using live data from all currently connected hosts. This inherently flushes out the stale cache and allows the previously partitioned host to be recognized and fully integrated.