ESXi hosts lose network connectivity on vSAN network when multiple active uplinks available in two node cluster with direct NIC to NIC connectivity.

Products

• VMware vSphere ESXi
• VMware vSAN

Issue/Introduction

• ESXi host has point to point network connectivity in Two Node VSAN ROBO cluster.
• vDS portgroup used for vSAN has load balancing configured as "Route Based on Port ID" and has all uplinks as active.
• After a reboot of ESXi host, vSAN network connectivity is not working between ESXi hosts.
• When network connectivity is not working between hosts over vSAN network/vmk you see through esxtop that ESXi1 is using differnet vmnic from ESXi2 in the esxtop. To check that:
  • Login to ESXi host over SSH session as root.
  • type esxtop and click enter.
  • Click on n for networking
  • Find vSAN vmk and notice the TEAM-PNIC assigned to it:
    

Cause

This is expected behavior when you have uplinks connected directly between ESXi hosts and using the load balancing method (like "Route Based on Port ID/Physical Nic Load") allowing vmk to pick only one uplink.

Resolution

If you have Multiple uplinks on vDS/vSwitch connected in a point to point connection between two ESXi hosts for vSAN communication, you can either:

• have one uplink as active and the second uplink as standby, this way vmk will always use one uplink unless it fails, then it will use the standby one:  Right click on vDS Portgroup -> click on Teaming and Failover 
  
  
  or
  
  
• have both uplinks as active but use "Route based on IP Hash" load balancing to create a static port channel so that vmk uses both uplinks at the same time: Right click on vDS Portgroup -> click on Teaming and Failover 
  

 

Additional Information

How to configure NIC teaming in ESXi and ESX