vSAN Cluster UUID not in sync between vCenter and ESXi

Products

• VMware vSAN

Issue/Introduction

Impact/Risks:
This can cause host partitions when adding hosts to the vSAN cluster via the vCenter Web Client
This can result in host not contributing performance stats
vSAN Health Service Alert vCenter state is authoritative

Symptoms:

• When adding a new host or re-adding a host to a vSAN cluster the host becomes network partitioned.

• vSAN Health Service Alert "vCenter state is authoritative".

• vSAN Health Service Warning "All Hosts Contributing Stats" especially if the entire cluster's vSAN UUID is out of sync
  
• This will also prevent enabling/re-enabling Performance Service on the cluster

To confirm the mismatch between vCenter and ESXi do the following:

1. In vCenter Web Client navigate to vSAN cluster  and go to Monitor tab. Under monitor tab, under vSAN section, click on Support  and you should see Obfuscation highlighted.

Note: The string of alphanumeric characters before the ':' and not highlighted is the VcUuid and also aligns to the Config Generation in the vSAN cluster get output as seen below in the next step. The string of alphanumeric characters after the ':' is the vSAN cluster UUID and should align with the Sub-Cluster UUID as seen in the below vSAN cluster get output.

SSH to one of the hosts in the cluster that is part of the vSAN cluster and run esxcli vsan cluster get and As you can see the vSAN cluster UUIDs don't match up. Refer the below snippets.

[root@host] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2022-08-05T22:21:32Z
Local Node UUID: 5c48d76f-07f0-fa60-81ac-############
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5c4f4d53-d623-d510-fb45-############
Sub-Cluster Backup UUID: 5c48d99b-573c-d974-1586-############
Sub-Cluster UUID: 52f1b2fb-fe25-3210-4690-############
Sub-Cluster Membership Entry Revision: 71
Sub-Cluster Member Count: 5
   Sub-Cluster Member UUIDs: 6684cf2f-3456-001f-d628-############, 6684cb9c-5d4d-56a5-e6f8-############, 6684cbc5-3343-1b39-b068-############, 6684cd07-2f5d-62e5-6cbe-############, 6684d8f2-0edd-fce3-58af-############
   Sub-Cluster Member HostNames: esxi24, esxi22, esxi21, esxi23, esxi25
   Sub-Cluster Membership UUID: 2539d366-7806-b26c-e9b3-############
   Unicast Mode Enabled: true
   Maintenance Mode State: OFF
   Config Generation: 429d3255-d614-46bf-91c3-############ 4 2024-09-01T04:26:32.0
   Mode: REGULAR
   vSAN ESA Enabled: false

Environment

VMware vSAN (All Versions)

Resolution

If the above symptoms and issue matches, please contact Broadcom Support to investigate the issue.