vSAN Skyline Health Reports “vSAN Cluster Partition” After Re-adding Reinstalled Host

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• The ESXi host operating system reinstalled due to a hardware failure.
• The host re-added to an existing vSAN cluster following the OS reinstallation.
• After rejoining, vSAN Skyline Health reports a "vSAN cluster partition" error.

• The newly added host appears as network partitioned.

Environment

• VMware vSAN 7.x
• VMware vSAN 8.x
• VMware vSAN 9.x

Cause

The network partition is caused by adding the affected host back to the vSAN cluster using a mismatched Sub-Cluster UUID.

Cause validation:

• Run the command "esxcli vsan cluster get". The output will confirm a mismatch in the Sub-Cluster UUID between the affected host and the healthy cluster members.
  
  Non working host : 
  esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTtt:mm:ssZ
   Local Node UUID: ########-####-####-####-############
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
     Sub-Cluster Master UUID: ########-####-####-####-############
   Sub-Cluster Backup UUID:
   Sub-Cluster UUID: ########-####-####-####-##########10
   Sub-Cluster Membership Entry Revision: 0
   Sub-Cluster Member Count: 1
  
  Working host :
  esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: YYYY-MM-DDTtt:mm:ssZ
   Local Node UUID: ########-####-####-####-############
   Local Node Type: NORMAL
   Local Node State: MASTER
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: ########-####-####-####-############
   Sub-Cluster Backup UUID:
   Sub-Cluster UUID: ########-####-####-####-##########11
   Sub-Cluster Membership Entry Revision: 1
   Sub-Cluster Member Count: 2
   
   

Resolution

• To resolve the vSAN cluster partition issue, the affected host must be properly rejoined to the vSAN cluster with the correct Sub-Cluster UUID. Follow the steps below:

  1. Remove the host from the vSAN cluster: Log in to the ESXi SSH session of the affected host and run the following command to remove it from the existing (incorrect) vSAN cluster configuration:

     localcli vsan cluster leave

  2. Rejoin the host to the correct vSAN cluster: Identify the correct sub-cluster UUID from a healthy cluster member by running esxcli vsan cluster get. Then, run the following command on the affected host to rejoin the cluster:

     localcli vsan cluster join -u <correct-Sub-Cluster-UUID>

  3. Verify cluster health: After the host has rejoined the cluster, check the vSAN Skyline Health service to ensure that the "vSAN cluster partition" health check is cleared and that the host is now properly contributing to the cluster.

• If the issue persists, verify the unicast agent list and update it if required by following the steps in the below mentioned knowledge base article.
  
  Configuring vSAN Unicast networking from the command line