vSAN Unicast Connectivity Check Fails After Firmware Upgrade Due to Missing Physical Uplinks

Products

• VMware vSAN

Issue/Introduction

Symptoms

After performing a firmware upgrade on a vSAN cluster, the following health check warnings were observed in Skyline Health:

• vSAN: Basic (Unicast) connectivity check failure on vmk1 <-- vSAN
• vSAN: MTU check (large packet ping) failure
• vSAN: Cluster Partition and Object Health errors
• Overall Skyline Health Score dropped to 39

Environment

VMware vSAN 7.x

Cause

The root cause of the issue was identified as the loss of physical uplinks on host. These NICs were either physically removed, failed, or not detected by the ESXi host.

The absence of these uplinks caused:

• Unicast communication failure between vSAN nodes
• Network partitioning, leading to object access degradation

Contributing observations:

• vmk1 (used for vSAN traffic) was mapped only to vmnic on impacted host, while peer hosts had 2 vmnics as active uplinks.
• ICMP pings from impacted host to peer hosts failed, indicating isolation.
• ILO - indicated PCI-E Slot (where NICs are expected) as N/A, whereas it was functional on other hosts.
• Kernel logs (/var/log/vmkernel.log) showed continuous CMMDSNetSendtoUnicastChannels errors and FCoE discovery failures on vmhbas.

Sample errors:

2025-07-27T18:51:36.238Z cpu53:2098643) CMMDSNet: CMMDSNetSendtoUnicastChannels: 1486: Throttled: 523925a0-####-####-####-1524a1d23f74: Failed to send to unicast host '192.###-###.23;12321' on iface '192.###-###.21': Host is down.
2025-07-27T18:51:39.746Z cpu73:2098270) WARNING: q1_fcoe:vmhba65: FipVlanTimeoutWork: 258: FIP VLAN Max Retries reached, cur vlan and pri: 1002
2025-07-27T18:51:42.033Z cpu0:2098280) WARNING: ql_fcoe:vmhba67: FipDiscoveryTimeoutWork: 126: Max retry exhausted for Fabric - 0x431#####b5b0 vlan_id - 0x3ea

These indicate hardware link failure and vSAN communication breakdown.

Resolution

Engage the hardware vendor to inspect host.

 

Additional Information

• A minimum of two physical NICs is recommended for redundancy in vSAN environments.
• MTU mismatches or missing NICs can directly impact vSAN health, performance, and availability.