vSAN Cluster UUID not in sync between vCenter and ESXi

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• When adding a new host/witness or re-adding a host to a vSAN cluster via vSphere Web client the host becomes network partitioned.

• The VvUuid is also out of sync on one or more hosts in the vSAN cluster.

• The vSAN Skyline Health Alert "vCenter state is authoritative" is triggered.

To confirm the mismatch of vSAN UUID between vCenter and ESXi do the following:

1. In vSphere Web Client navigate to vSAN cluster  and go to Monitor tab. Under Monitor tab, under vSAN section, click on Support  and you should see Obfuscation highlighted:
   
   Note: The string of alphanumeric characters before the ':' and not highlighted is the VcUuid and also aligns to the Config Generation in the esxcli vsan cluster get output as seen below in the next step. The string of alphanumeric characters after the ':' is the vSAN cluster UUID and should align with the Sub-Cluster UUID as seen in the below esxcli vsan cluster get output.
   
   

2. SSH to one of the hosts in the cluster that is part of the vSAN cluster (not partitioned) and run esxcli vsan cluster get , as seen below the vSAN cluster UUIDs don't match up with what is seen on the vSphere Client. Refer the below snippets.

   [root@host] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: YYYY-MM-DDTHH:MM:SSZ
Local Node UUID: 5c48d76f-07f0-fa60-81ac-############
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5c4f4d53-d623-d510-####-############
Sub-Cluster Backup UUID: 5c48d99b-573c-d974-####-############
Sub-Cluster UUID: 52f1b2fb-fe25-3210-####-############
Sub-Cluster Membership Entry Revision: 71
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 36979731-f1b3-4ea6-####-############ 166 YYYY-MM-DDTHH:MM:SS.SSS
Mode: REGULAR

Environment

VMware vSAN 7.0.x

VMware vSAN 8.0.x

Cause

• One of the ways the vSAN UUID goes out of sync between vCenter and vSAN is:

  • When the vSAN cluster is "Turned OFF" instead of "Shutdown".

  • To bring vSAN back ON, the hosts are joined back to the vSAN cluster as per Improper vSAN Shutdown and Accidental turning off vSAN leads to Cluster Configuration Loss and Service Disruption.

  • Now the new vSAN cluster would have a different vSAN cluster UUID on vSphere client, while the hosts would have a different vSAN cluster UUID.

Resolution

If the above symptoms and issue matches, please contact Broadcom Support to investigate the issue.