VSAN health check failed: vSAN cluster partition when trying to patch stretched cluster to ESXI 8.0U3d

Products

• VMware vSAN

Issue/Introduction

Attempting to patch or upgrade a vSAN stretched cluster and the process is failing due to vSAN health check "vSAN Cluster Partition" failing.

esxcli vsan cluster unicastagent list dispalys all entries correctly.

esxcli vsan cluster unicastagent list
NodeUuid                                 IsWitness  Supports Unicast  IP Address  Port  
661e5ede-506f-f314-837b-################    0        true             10.#.#.#   12321
e99db500-f2c4-4e49-b45c-################    0        true             10.#.#.#   12321
66aceba8-6b7a-1a6d-7e2f-################    1        true             10.#.#.#   12321

esxcli vsan cluster get shows the witness host missing.

esxcli vsan cluster get
Cluster Information
   Enabled: true
   Current Local Time: 2025-04-02T12:46:53Z
   Local Node UUID: 661e5ede-506f-f314-837b-################
   Local Node Type: NORMAL
   Local Node State: BACKUP
   Local Node Health State: HEALTHY
   Sub-Cluster Master UUID: e99db500-f2c4-4e49-b45c-################
   Sub-Cluster Backup UUID: 661e5ede-506f-f314-837b-################
   Sub-Cluster UUID: ########-####-####-####-############
   Sub-Cluster Membership Entry Revision: 3
   Sub-Cluster Member Count: 2
   Sub-Cluster Member UUIDs:
     661e5ede-506f-f314-837b-################, e99db500-f2c4-4e49-b45c-################

The hosts can ping the witness, and the witness can ping the hosts over the correct vmkernel ports for witness traffic.

 

Environment

ESXi/vSAN upgrading to 8.0 Update 3

Cause

vSAN heartbeat/clustering port 12321 is blocked between the witness and cluster hosts.

Port 12321 is required to be open between all vSAN hosts and witness for cluster formation and membership.

tcpdump performed between the cluster leader node and the witness, grepping for 12321 shows that traffic using this port is only traversing the network in one direction.

Using tcpdump-uw on witness node grep for port 12321, receiving from cluster leader and backup nodes, and replying, however on the cluster leader and backup nodes grepping witness IP and port 12321 shows sending to, but receiving no reply from the witness.

Resolution

Work with your network security team which manages any firewalls or security appliances to ensure traffic is allowed bidirectionally for all required vSAN ports.

Additional Information

See the following for additional details:

• vSAN Ports
• vSAN Health Service - Network Health - vSAN Cluster Partition
• vSAN Witness appliance partitioned from the stretched cluster