Whether this connection can be successfully established between two nodes can be validated by running one node in server mode and connecting to it via another node:
On node1:
# openssl s_server -cert /etc/vmware/ssl/rui.crt -key /etc/vmware/ssl/rui.key -port 12443
On node2:
# openssl s_client -connect <vsan-ip-of-node1>:12443 -cert /etc/vmware/ssl/rui.crt -key /etc/vmware/ssl/rui.key
If the above works without issue but enabling vSAN Data in Transit encryption is still failing then validate the stored unicastagent list data vs the nodes actual current cert information e.g.:
On node1 (check the cert thumbprint info for node2):
# esxcli vsan cluster unicastagent list
Get the FQDN of node2 from the node UUIDs:
# cmmds-tool find -t HOSTNAME
On node2, check the current cert thumbprint and that it matches what is in unicastagent list of the other nodes:
# openssl x509 -in /etc/vmware/ssl/rui.crt -noout -sha1 -fingerprint