vSAN Cluster Partition Due to Bidirectional Traffic Blocking on vSAN Heartbeat Ports

Products

VMware vSAN

Issue/Introduction

Symptoms

Skyline Health reports "vSAN Cluster Partition."

esxcli vsan cluster unicastagent list shows cluster members, but traffic is blocked.

esxcli vsan cluster unicastagent list
NodeUuid                                 IsWitness  Supports Unicast  IP Address  Port  
661e5ede-506f-f314-837b-################    0        true             10.#.#.#   12321
e99db500-f2c4-4e49-b45c-################    0        true             10.#.#.#   12321
66aceba8-6b7a-1a6d-7e2f-################    1        true             10.#.#.#   12321

esxcli vsan cluster get indicates a missing or isolated host.

esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2025-04-02T12:46:53Z
   Local Node UUID: 661e5ede-506f-f314-837b-################
   Local Node Type: NORMAL
   Local Node State: BACKUP
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: e99db500-f2c4-4e49-b45c-################
   Sub-Cluster Backup UUID: 661e5ede-506f-f314-837b-################
   Sub-Cluster UUID: ########-####-####-####-############
   Sub-Cluster Membership Entry Revision: 3
   Sub-Cluster Member Count: 2
   Sub-Cluster Member UUIDs:
     661e5ede-506f-f314-837b-################, e99db500-f2c4-4e49-b45c-################

The hosts can ping the witness, and the witness can ping the hosts over the correct vmkernel ports for witness traffic.

Environment

VMware vSAN 8.x

VMware vSAN 7.x

Cause

vSAN cluster operations rely on UDP port 12321 and TCP port 2233 for heartbeat and membership communication. The cluster partitions if network security policies or configuration changes block this traffic bidirectionally.

tcpdump-uw on port 12321 shows traffic is blocked in one direction - tcpdump performed between the cluster leader node and the witness, grepping for 12321 shows that traffic using this port is only traversing the network in one direction.

[root@ESXi:~ ] pktcap-uw -- vmk <Witness tagged vmk e.g., vmk5> -- dir 2 -o - | tepdump-uw -enr - | grep -i 12321
The name of the vmk is vmk5.
pktcap: The output file is -.
pktcap: No server port specifed, select 64673 as the port.
pktcap: Local CID 2.
pktcap: Listen on port 64673.
pktcap: Main thread: 953112226624.
pktcap: Dump Thread: 953112762112.
pktcap: Recv Thread: 953113290496.
reading from file -, link-type EN10MB (Ethernet)
pktcap: Accept ...
pktcap: Vsock connection from port 1025 cid 2.
12:27:14.483767 00:50:56:6a: 4f : 4f > cc:d8 : 1f : c4:2b:19, ethertype IPv4 (0x0800), length 482: <Host IP>.12321 > <Witness IP>.12321: UDP, length 440
12:27:15.483768 00:50:56:6a: 4f : 4f > cc:d8 : 1f : c4:2b:19, ethertype IPv4 (0x0800), length 482: <Host IP>.12321 > <Witness IP>.12321: UDP, length 440

Using tcpdump-uw on witness node grep for port 12321, receiving from cluster leader and backup nodes, and replying, however on the cluster leader and backup nodes grepping witness IP and port 12321 shows sending to, but receiving no reply from the witness.

Resolution

Verify Ports: Ensure bidirectional traffic is allowed for UDP port 12321 and TCP port 2233.
Network Configuration: Work with the network security team to allow bidirectional traffic for all required vSAN ports.

Additional Information

Consult the Broadcom Ports Documentation to validate all required vSAN port configurations.

See the following for additional details: