Configuring vSAN Unicast networking from the command line

Products

VMware vSAN

Issue/Introduction

This article provides step-by-step instructions on how to manually configure vSAN Unicast networking from esxcli.

Note: vSAN Unicast mode is a feature that was introduced in vSAN 6.6 (vSphere 6.5.0d)
Unicast replaced legacy multicast networking in order to simplify the network requirements of maintaining a vSAN cluster's Quorum.

1. Occasionally when upgrading vSAN Clusters to vSphere 6.5.0d or later, one or more hosts may fail to inherit the Cluster Member Updates from vCenter.

2. In some cases, the list of vSAN nodes maintained by vCenter and the number of actual nodes may differ due to nodes being added or removed while vCenter was unavailable.

Impact/Risks:

Network partitioning is a risk if the Unicast agent list is not properly rebuilt. In most situations, the vSAN cluster has already formed multiple partitions. The goal of rebuilding the table is to form one vSAN cluster.

Note: When building a Unicast agent list on an ESXi host, add entries for all the other hosts, but never add the IP of the host whose table is being configured.

When a vSAN Data Node or Witness Node has its own IP address in its Unicast agent list, many networking problems can arise and could lead to a PSOD.

Environment

VMware vSAN 8.x
VMware vSAN 7.x
VMware vSAN 6.x

Cause

Resolution

You can use the following steps to manually rebuild the Unicast agent list on each host:

1. Identifying necessary vSAN Cluster information.
2. Building the vSAN Unicast agent list.
3. Last Steps.

1. Identify the VMkernel port used for vSAN, its IP address, and the node UUID on each host in the cluster.

1.1 SSH to every ESXi host in the cluster and login as root.

1.2 Run the following command to identify the VMkernel port used for vSAN, and copy the output for later use:

esxcli vsan network list

[root@server name:~] esxcli vsan network list
Interface
VmkNic Name: vmk1
IP Protocol: IP
Interface UUID: ########-####-####-####-############
Agent Group Multicast Address: 224.2.3.4
Agent Group IPv6 Multicast Address: ff19::2:3:4
Agent Group Multicast Port: 23451
Master Group Multicast Address: 224.1.2.3
Master Group IPv6 Multicast Address: ff19::1:2:3
Master Group Multicast Port: 12345
Host Unicast Channel Bound Port: 12321
Data-in-Transit Encryption Key Exchange Port: 0
Multicast TTL: 5
Traffic Type: vsan

Note: Take note of the VmkNic Name - in the above output it's "vmk1"

1.3 Find the IP address for "vmk1" with the following command:

esxcli network ip interface ipv4 get | grep vmk1

[root@server name:~] esxcli network ip interface ipv4 get | grep vmk1
vmk1 ###.##.##.## ###.##.###.### ###.##.###.### STATIC ###.##.###.### false

Note: The IP address of vmk1 on this host is: ###.##.##.##

1.4 Find the node UUID of the host with the following command:

cmmds-tool whoami

[root@server name:~] cmmds-tool whoami
########-####-####-####-############

1.5 Equipped with host UUID and vSAN VMkernel port IP address for ALL hosts in the cluster, start building the Unicast agent lists.

3 Node cluster example:

2. Building the Unicast Agent List

2.1 Before making changes to the Unicast agent lists, run the following command on all nodes in the cluster to temporarily ignore "Cluster Member List Updates" coming from vCenter.

esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates

[root@server name:~] esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListupdates
Value of IgnoreClusterMemberListUpdates is 1

2.2 Sometimes, there might be incorrect entries in a host's Unicast agent list such as:

- "Supports Unicast" is set to False for a particular host, an incorrect IP address, an incorrect Node UUID, or a host which is NOT a Stretched Cluster Witness host is incorrectly marked with "IsWitness" as True.
- An existing entry cannot be modified. It must be deleted and re-added.

2.2.1 Check the current unicast agent list with the following command:

esxcli vsan cluster unicastagent list

[root@server name:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address
------------------------------------ --------- ---------------- -------------
########-####-####-####-############ 0 false ###.##.###.##
########-####-####-####-############ 0 false ###.##.###.##

Note: In the output above, the second entry has an incorrect IP address, and both entries have the "Supports Unicast" flag set as "false".

Note: For encrypted environments, take note of the thumbprint as it appears here. It should be the same for all nodes since it comes from the vCenter. It will need to be added to the new entries.

2.2.2 To fix this problem, run the following command to delete the errant entries. You can also use this to delete all the entries and rebuild the unicast agent list from scratch.

Syntax: esxcli vsan cluster unicastagent remove -a <Host_VSAN_IP>

[root@server name:~] esxcli vsan cluster unicastagent remove -a ###.##.###.##

Note: By running esxcli vsan cluster unicastagent list we can verify if the entry is cleared.

2.3 Add entries to the unicast agent list.

- **REMINDER** When building the Unicast agent list on an ESXi host, add entries for all the other hosts but never add the IP of the host whose table is being configured.
- When an ESXi host has its own IP address in its Unicast agent list, vSAN can go unstable, networking problems can arise and potentially lead to the host encountering a PSOD.
- Using the 3 node example, each host will have 2 entries.

Data Node entry syntax:

esxcli vsan cluster unicastagent add -t node -u <Host_UUID> -U true -a <Host_VSAN_IP> -p 12321

Example on host 1:

[root@server name1:~]esxcli vsan cluster unicastagent add -t node -u ########-####-####-####-############ -U true -a ###.##.###.## -p 12321

[root@server name1:~] esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint SubClusterUuid
------------------------------------ --------- ---------------- -------------- ----- ---------- ----------------------------------------------------------------------------------------------- --------------
########-####-####-####-############ 0 true ###.##.###.## 12321 ########-####-####-####-############

Note: After running the commands, check the Unicast agent list to confirm entries were added correctly as shown in the output above.

Note: in a Stretched Cluster, you must set the "IsWitness" flag to "True" for the Witness host entry.

Note: For encrypted environments, you must add the vCenter's thumprint with the -T switch as part of the command string.

Witness Node entry syntax:

esxcli vsan cluster unicastagent add -t witness -u <Host_UUID> -U true -a <Host_VSAN_IP> -p 12321

2.4 Repeat from step 2.1 on the remaining vSAN hosts, making sure to not include the IP of the host whose table is being configured.

3. Last Step.

3.1 Assuming there are no issues with the physical network, the vSAN Cluster should form right away.
If the rebuild on each host was done correctly, the correct number of Cluster members should be shown in the following command,

esxcli vsan cluster get

[root@server name:~] esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2024-08-16T06:22:52Z
Local Node UUID: ########-####-####-####-############
Local Node Type: NORMAL
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: ########-####-####-####-############
Sub-Cluster Backup UUID: ########-####-####-####-############
Sub-Cluster UUID: ########-####-####-####-############
Sub-Cluster Membership Entry Revision: 37
Sub-Cluster Member Count: 2 ←--------------
Sub-Cluster Member UUIDs: ########-####-####-####-############, ########-####-####-####-############
Sub-Cluster Member HostNames: server name1, server name2
Sub-Cluster Membership UUID: ########-####-####-####-############
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: ########-####-####-####-############ 7 2024-08-16T06:07:52.230
Mode: REGULAR

3.2 After all the process has been completed successfully, make sure to enable the "Cluster Member List Updates" again on all the nodes in the cluster. Run the following command:

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates

[root@server name:~] esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListupdates
Value of IgnoreClusterMemberListUpdates is 0

Additional Information

If you find that you have completed your upgrade to vSAN 6.6 including the on-disk upgrade process and your hosts are still utilizing multicast for cluster communications, confirm that you do not have an orphaned/dead host information stuck in cmmds as this can prevent the conversion from occurring naturally.

VMware does not support reverting back to multicast mode after the vSAN 6.6 installation or upgrade has completed and unicast mode has been successfully configured.