After adding a new host to the vSAN cluster and removing another, we are now encountering a "Partitioned Cluster" error.

Products

• VMware vSAN

Issue/Introduction

When you add a new host that is on a higher build/version to the vSAN cluster, when the ESXi versions are not uniform in the cluster (all hosts on same build). This can create a "vSAN Cluster Partition" situation in that vSAN cluster. If you are actively doing an upgrade of the ESXi hosts in the vSAN cluster, finish the upgrades before adding a new host. 

When mixed ESXi versions are present, the new host may cause the cluster to split into separate sub-clusters (e.g., Partition 1 and Partition 2), resulting in virtual machine (VM) inaccessibility.

Environment

vSAN 8.0 or later

Cause

The cluster partition is caused where a new host is added when the cluster is running mixed ESXi versions during an active upgrade. This creates an unintentional sub-cluster formation as the new host attempts to join the vSAN network with the newest version, when nodes on the lower version remain. 

Best practice dictates that upgrades should be completed on all nodes before introducing new hosts to the cluster.

Resolution

1. SSH into all hosts on the cluster. Alternatively, you only need the vSAN Master host and the affected host (but this will ensure consistency for clusters of 6-nodes or smaller). 
2. Verify the unicast table entries on hosts (specifically the master and impacted, which is the host representing the majority of the original cluster) to ensure it recognizes the missing hosts (do this from all hosts, there should be all other vSAN hosts, other than itself (ie 4 node cluster, would have three entries, where the entries are the other hosts). 
   
   • esxcli vsan cluster unicastagent list
     •  If entries are missing, see KB: Rebuilding unicast tables from CLI
3. Run the command to verify cluster membership of the cluster to ensure it matches the remaining hosts (confirm 'Subcluster UUID' matches on all hosts). Copy this ID and use in Step 6.  
   • esxcli vsan cluster get
4. If you see VMs showing online on the impacted host, power them down. (May have to be 'forced' down, if not responding to from vCenter). 
5. After all VMs are offline, remove the isolated host from the incorrect cluster membership by running:
   • esxcli vsan cluster leave
6. Then you will rejoin the host back into the master vSAN cluster membership using the following command (replace <subclusterUUID> with the UUID of the master cluster; this can be retrieved using the command in step 3: esxcli vsan cluster get):
   • esxcli vsan cluster join -u <subclusterUUID>
7. After you have rejoined the impacted host into the cluster, verify the 'Subcuster Member Count' has increased to match the number of hosts. 
   • esxcli vsan cluster get
8. Allow vSAN to balance out and sync the hosts rejoining the cluster. Where you can monitor the resync under the Cluster -> Monitor -> Resyncing Objects. 

 

*If assistance is needed in this process, please contact Broadcom Support. 

Additional Information

• For future upgrades, verify host space consumption prior to beginning the upgrade process. If capacity is critical (e.g., >95% consumption), add additional hosts to the cluster to accommodate consumption requirements before starting the upgrade.
• Then complete the upgrade process for all existing hosts within the cluster before introducing new nodes to ensure compatibility and cluster stability.

References: 

1. vSAN Cluster Partition
2. Rebuilding unicast tables from CLI. 
3. Contact Broadcom Support.