vSAN Cluster UUID not in sync between vCenter and ESXi

Products

• VMware vSAN

Issue/Introduction

Impact/Risks:
This can cause host partitions when adding hosts to the vSAN cluster via the vCenter Web Client
This can result in the VcUuid also to be out of sync on one or more hosts in the cluster.
vSAN Skyline Health Alert vCenter is not authoritative

Symptoms:

• When adding a new host or re-adding a host to a vSAN cluster the host becomes network partitioned.

• vSAN Skyline Health Alert "vCenter state is authoritative".

To confirm the mismatch between vCenter and ESXi do the following:

1. In vCenter Web Client navigate to vSAN cluster  and go to Monitor tab. Under monitor tab, under vSAN section, click on Support  and you should see Obfuscation highlighted.

Note: The string of alphanumeric characters before the ':' and not highlighted is the VcUuid and also aligns to the Config Generation in the vSAN cluster get output as seen below in the next step. The string of alphanumeric characters after the ':' is the vSAN cluster UUID and should align with the Sub-Cluster UUID as seen in the below vSAN cluster get output.

SSH to one of the hosts in the cluster that is part of the vSAN cluster and run esxcli vsan cluster get and As you can see the vSAN cluster UUIDs don't match up. Refer the below snippets.

[root@host] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2022-08-05T22:21:32Z
Local Node UUID: 5c48d76f-07f0-fa60-81ac-############
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5c4f4d53-d623-d510-fb45-############
Sub-Cluster Backup UUID: 5c48d99b-573c-d974-1586-############
Sub-Cluster UUID: 52f1b2fb-fe25-3210-4690-############
Sub-Cluster Membership Entry Revision: 71


Environment

VMware vSAN 7.0.x
VMware vSAN 8.0.x

Resolution

If the above symptoms and issue matches, please contact Broadcom Support to investigate the issue.