vSAN cluster partition due to invalid unicast agent list

Products

VMware vSAN

Issue/Introduction

In vSAN unicast mode (vSAN 6.6 and later), each ESXi host maintains a Unicast Agent list to facilitate cluster communication. If this list becomes stale, invalid, or incomplete, the cluster may experience network partitions, even if basic network connectivity (ping) between vSAN VMkernel ports remains functional.

VMs show as invalid or inaccessible on host.
vSAN Skyline Health is showing a vSAN Cluster Partition Alert
Command line verification for the vSAN cluster health show.
esxcli vsan health cluster list
The output shows that the Health Test "vSAN cluster partition" is showing with status "red"

Example:

esxcli vsan health cluster list
Health Test Name Status
-------------------------------------------------- ------
Network Red
vSAN cluster partition Red

Upon validating further using command esxcli vsan health cluster get -t clusterpartition or esxcli vsan health cluster get -t "vSAN cluster partition" the output show the hosts in different partitions.

[root@localhost:~] esxcli vsan health cluster get -t clusterpartition

vSAN cluster partition RedChecks if the vSAN cluster is partitioned due to a network issue.

Hosts list
+-----------+-----------+------------------------------+
| Host | Partition | Host UUID |
+-----------+-----------+------------------------------+
| :#.#.#.69 | 1 | 699584e0-2b07-12e3-2806-#### |
| :#.#.#.70 | 1 | 6995864d-ae5e-49f7-b8f1-#### |
| :#.#.#.71 | 1 | 699660b7-9694-84ad-aea5-#### |
| :#.#.#.68 | 2 | 6997c4fe-153d-c3a5-5835-#### |
+-----------+-----------+------------------------------+

Post adding a new Host to the vSAN Cluster, the alert "Host cannot communicate with one or more other nodes in the vSAN enabled cluster" is shown on the Summary Tab of all vSAN Hosts:
Hosts vSAN vmkernel IP addresses are changed to new IP addresses. However, the unicast table on hosts contain old IP addresses.

Environment

VMware vSAN (6.6 and later).

Cause

This issue occurs when the vCenter Server fails to automatically update the unicast agent list on ESXi hosts following a direct change to the cluster such as a IP change on the vSAN VMkernel interfaces, Host add, other events, the cluster may partition even if physical connectivity is verified.

Examples:

The below output for 4 node vSAN cluster show one of the hosts missing unicast entry.

[root@esxi-03:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid
------------------------------------ --------- ---------------- ------------- ----- ---------- ----------------------------------------
602572bd-2ef4-8f69-d8ce-############ 0 true ###.###.10.13 12321
60257046-5d95-a750-7135-############ 0 true ###.###.10.11 12321
602583eb-233c-b69a-8291-############ 0 true ###.###.10.12 12321

[root@esxi-04:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid
------------------------------------ --------- ---------------- ------------- ----- ---------- ----------------------------------------
602572bd-2ef4-8f69-d8ce-############ 0 true ###.###.10.13 12321
60198995-b367-2922-8fbf-############ 0 true ###.###.10.14 12321

The vSAN vmkernel IP configured in the vmk3 different from the network show up in the unicast entries.

[root@host-1:~] esxcfg-vmknic -l
Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS
vmk3 vSAN IPv4 10.##.##.## ##:##:##:## 10.##.##.## ##:##:##:##:##:## 9000 65535 true

[root@host-1:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port
------------------------------------ --------- ---------------- ------------ -----
68d54366-9e30-eb05-f202-#### 1 true 10.##.##.## 12321
68cd2243-7e11-8499-469a-#### 0 true 172.##.##.## 12321

Resolution

In order to resolve the issue, fix the unicastagent list to match with appropriate host UUID, vSAN vmkernel IP and certificate thumbprint.

Step-1: Prepare the hosts:

Before making any changes to the Unicastagent list via ESXi CLI IgnoreClusterMemberListupdates needs to be set to 1 on all hosts to avoid any updates from vCenter server.
To set to value of "1": esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
(NOTE: "In ESXi 9.1 and later, strict SHA-256 thumbprint enforcement is applied to vSAN cluster communication. If /VSAN/IgnoreClusterMemberListUpdates is set to 1, hosts may retain legacy SHA-1 thumbprints, causing a total network partition and 0 TB reported capacity after an upgrade, as ESXi 9.1 will reject non-SHA-256 thumbprints.)

Step-2: Gather the required details for all the hosts using command line.

Host UUID: Find the UUID for the Host using command below. Collect it for all the required hosts.
[root@host-1:~] cmmds-tool whoami
60257046-5d95-a750-7135-############
Collect the vsan vmkernel IP for the required hosts.
1. Run command esxcli vsan network list to capture the vmkernel enabled with vSAN network.
2. Run the command esxcli network ip interface ipv4 get or esxcfg-vmknic -l to get the IP associated with the vmkernel adapter configured with vSAN network.
Collect certificate information for the hosts required using the command below.

[root@host-1~] openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha1 -noout
sha1 Fingerprint=##:##:##:##

Step-3: Update the unicast table for the affected hosts using:

Warning: A vSAN host unicast table must contain the peer details of all other neighbor nodes in the cluster and must never contain its own local IP or UUID. Including a host's own details in its unicast table is an invalid configuration that can break cluster communication or cause critical networking faults.

Add the missing entry to the Unicast Agent list ( Reference ) on all Host(s) where it is required using the syntax below.

esxcli vsan cluster unicastagent add -t node -u <Host_UUID> -U true -a <Host_VSAN_IP> -p 12321 -T <Host Cert Thumbprint>
Verify that the unicast table on all the hosts to contain neighbor hosts details. Use command esxcli vsan cluster unicastagent list on all the hosts to validate.

Step-4: Validate cluster member count:

Run command esxcli vsan cluster get on hosts to validate the cluster is not in network partition state.

Step-5: Revert the advanced parameters modified during the activity.

Once all changes have been made to the Unicastagent list on all impacted hosts set IgnoreClusterMemberListupdates back to default value of 0.

To set to value of "0": esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates

If assistance is required, please open a Ticket with VMware by Broadcom Support.