vSAN Cluster went down post vLAN config changes

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• vSAN cluster may report host Network partition. 
• RDT handshake events observed in logs .

 vmkernel.log :

2025-03-26T06:31:23.732Z In(182) vmkernel: cpu58:2099293)RDTTCPConn: RDTTCPConnCheckEncrKey:236: 0x432a832e1e##(0x0): Failed to find encryption node context on server side during RDT handshake for node 67179c75-####-be89-####-d404e6dc15##
2025-03-26T06:31:24.306Z In(182) vmkernel: cpu58:2099293)RDTTCPConn: RDTTCPConnCheckEncrKey:236: 0x432a832e1e##(0x0): Failed to find encryption node context on server side during RDT handshake for node 67178d14-####-22a3-####-d404e6dc45##

Environment

VMware vSAN 8.x

Cause

Unicast table entries missing and Thumbprint missing on multiple hosts.

The following output displays the missing thumbprint details in the Unicast table

[root@Esx7] esxcli vsan cluster unicastagent list
+--------------------------+-----------+------------------+---------------+-------+---------------------------+
|         NodeUuid         | IsWitness | Supports Unicast |  IP Address   | Port  |      Cert Thumbprint      |
+--------------------------+-----------+------------------+---------------+-------+---------------------------+
| 671793##-7f2#-b03#-66##- |         0 | true             | 192.168.###.# | 12321 | ##:5F:BE:46:78:< >:##:##  |
| 6717a7##-e39#-644#-0a##- |         0 | true             | 192.168.###.# | 12321 | <Missing Thumbprint>      |
| 67179c##-c09#-be8#-2e##- |         0 | true             | 192.168.###.# | 12321 | ##:96:1C:9D:FD:< >:##:##  |
| 6717a2##-84a#-ae9#-ac##- |         0 | true             | 192.168.###.# | 12321 | ##:34:42:CD:BA:< >:##:##  |
| 67178d##-6e1#-22a#-64##- |         0 | true             | 192.168.###.# | 12321 | ##:F8:8A:E7:00:< >:##:##  |
| 6717b8##-8ce#-1de#-08##- |         0 | true             | 192.168.###.# | 12321 | ##:55:91:6D:E0:< >:##:##: |
+--------------------------+-----------+------------------+---------------+-------+---------------------------+

Unicast table on hosts missing when running below command to check on hosts

[root@Esx1:~] esxcli vsan cluster unicastagent list 
[root@Esx1:~]

Resolution

Add missing unicast entries on the hosts showing empty list with the correct thumbprint on all hosts.

To find the cert thumbprint on the host, SSH into the host itself and run the following command

root@Esx1~] openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha256 -noout
sha256 Fingerprint=##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:##:#

The below command can be used to update the unicast table on the hosts.

esxcli vsan cluster unicastagent add -t node -u <Host_UUID> -U true -a <Host_VSAN_IP> -p 12321 -T <Host Cert Thumbprint>