vSAN stretched cluster partition due to witness connectivity

Products

• VMware vSAN

Issue/Introduction

Symptoms :

• MTU misconfiguration may appear between Data node /Witness .
• Port configuration issues may occur during 2-node VxRail cluster setup, preventing successful deployment. The Witness Tag appears to be Missing on vSAN Data Nodes.
• vSAN connectivity fails between Data and Witness nodes, indicated by no traffic on port 12321, an MTU mismatch, and missing Witness traffic tagging on the Data node’s Management network .

• Verify network connectivity from the vSAN Witness Appliance to the data nodes over port 12321 by executing the following command directly on the Witness Appliance:

  tcpdump-uw -i vmkX | grep 12321

  And Execute the following command directly on the Primary and Backup nodes:

  tcpdump-uw -i vmkX | grep <witness IP/FQDN>

  Successful network connectivity is verified by observing bidirectional traffic comprising both incoming requests and corresponding responses over port 12321 on both the Primary and Backup nodes.

  When testing connectivity to TCP port 2233 between the data nodes and vSAN witness appliance, you will see an output similar to:

  # nc -z 172.xx.xx.xxx 2233
Connection to 172.xx.xx.xxx 2233 port [tcp] succeeded!

Environment

VMware vSAN 7.x

VMware vSAN 8.x

Cause

• Connectivity checks fail to detect traffic on TCP port 12321 flowing bidirectionally between the data nodes and the witness node.

0*:4*:1*.90******:08:e*:f*:f*:90 > 0*:5*:5*:8*:0*:e*, ethertype IPv4 (0*08**), length 282: 10.**.**.15.12321 > 10.**.**.**.12321: UDP, length 240
0*:4*:1*.90******:08:e*:f*:f*:90 > 0*:5*:5*:8*:0*:e*, ethertype IPv4 (0*08**), length 282: 10.**.**.15.12321 > 10.**.**.**.12321: UDP, length 240
0*:4*:1*.90******:08:e*:f*:f*:90 > 0*:5*:5*:8*:0*:e*, ethertype IPv4 (0*08**), length 282: 10.**.**.15.12321 > 10.**.**.**.12321: UDP, length 240
0*:4*:1*.90******:08:e*:f*:f*:90 > 0*:5*:5*:8*:0*:e*, ethertype IPv4 (0*08**), length 282: 10.**.**.15.12321 > 10.**.**.**.12321: UDP, length 240

• The bidirectional ping test with an MTU of 9000 fails or shows significant packet loss between the data and witness nodes.

[root@esxi:~] vmkping -I vmk2 10.2**.1*.3* -d -s 8972
PING 10.2**.1*.3* (10.2**.1*.3*): 8972 data bytes

--- 10.2**.1*.3* ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Resolution

In case Data nodes and the witness node are configured with mismatched MTU settings, along with improperly separated network traffic for witness communication and data services on the data nodes, it requires network reconfiguration as follows:

1.Separate witness and data traffic by assigning them to distinct VMkernel interfaces on the data nodes.

2.In case the data nodes and the witness nodes are using the same vmkernel interface for vSAN and witness traffic ,then the MTU shall be same across all the data and the witness node .

3.In case the data nodes and witness have different MTU configuration like data nodes have MTU as 9000 and vSAN witness nodes have MTU set as 1500 ,then the traffic shall be separated on the vSAN data nodes and witness traffic shall be tagged on different vmkernel interface in order to match the vSAN witness MTU configuration .

4.The MTU settings must be consistent across all interfaces involved in witness and data traffic.

5.Each VMkernel interface is correctly tagged and connected to the appropriate port group or VLAN to prevent cross-traffic and MTU-related issues.

6.Configure the vSAN witness tag on the management network of the data nodes using the below command:

• esxcli vsan network ip add -i vmk0 -T=witness  

7.Configure vSAN traffic tagging on the witness host’s management network using an MTU value that matches the vSAN MTU setting on the data nodes. This ensures successful bidirectional communication between the witness and data nodes.