Direct connect vSAN 2-node cluster is network partitioned after NIC replacement

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• Post Network Interface Card (NIC) replacement one or both the data nodes (ESXi host) in the cluster, one of the data nodes becomes partitioned from the cluster.

• The below vSAN Skyline alerts are triggered:

  1. vSAN cluster partition.

     • One data node and the witness host is in one partition, and the other data node is in another partition.

  2. vSAN: Basic (unicast) connectivity check

     • It shows basic ping failure between the two data nodes.

  3. vSAN : MTU check (ping with large packets)

     • It shows Jumbo frame ping failure between the two data nodes.

  4. vSAN Object Health

     • All objects in the vSAN cluster are in the "Reduced availability with no rebuild" state.

• Both data nodes use the same vmnics in Active/Standby configuration for the vSAN vmkernel adapter.

Environment

• VMware vSAN 7.x

• VMware vSAN 8.x

Cause

• Incorrect NIC cabling following the replacement caused the vSAN network failure between the two nodes.

• Since the two data nodes are connected via direct connect cables, it is important that the active NICs for vSAN on each host are connected to each other and similarly standby NIC on one host should be connected to the standby NIC on the other host.

• If the standby NIC on one node is connected to the active NIC on the other data node, communication over the vSAN network will not work. Thus causing the vSAN network partition.

• Example:

  • vmk2 on the data nodes are marked for vSAN traffic and has vmnic2 as active uplink and vmnic3 as standby.

  • Prior to the NIC replacement, vmnic2 of each host were connected to each other. Similarly vmnic3 of each host were connected to each other.

  • Post NIC replacement, vmnic2 of one host is connected to vmnic3 of the other host, and similarly the other port on the NIC was connected in a criss-cross fashion (vmnic3 to vmnic2).

  • Since the active uplinks on the hosts are not connected to each other, it would cause vSAN network connectivity to fail and thus partition the cluster.

Resolution

.To resolve this issue, either:

1. Change the active and standby uplink on one host:

   1. Navigate to vSphere Client > affected vSAN cluster > select the data node which is partitioned > Configure > Virtual Switches.

   2. Here, swap the active uplink with the standby uplink for the vSAN vmkernel adapter on the virtual switch. Configure NIC Teaming & Failover. (Perform this task only on one host.)

   3. Once this is done, the vSAN network connectivity should be restored between the two data nodes.

2. Else, change the physical cabling to match the active uplink configuration for the vSAN vmkernel adapter.