Routing Limitations when using RDMA for physical networking under vSAN

Products

Issue/Introduction

Customers utilizing RDMA with vSAN may encounter issues when using certain NIC teaming policies, such as "route based on physical network adapter load." This is due to the inherent design of the RDMA stack and how it handles network connections across multiple uplinks/RNICs.

Environment

vSAN 8.x

Cause

The RDMA stack does not support NIC teaming policies like "route based on physical network adapter load," IP-based hash, or LACP. This is an expected behavior because RDMA connections cannot spread across two uplinks/RNICs for the same vmk#.

Resolution

To ensure proper functionality of vSAN with RDMA, customers must use supported NIC teaming policies. The currently supported teaming policies for RDMA are:

1.  "Route based on the originating virtual port"
2.  "Route based on source MAC hash"
3.  "Use explicit failover order"
4.  Teaming override

 

Additional Information

This information is consistent with existing documentation on designing the vSAN network. For more details on vSAN network design and supported configurations, please refer to the official VMware vSAN documentation:

  https://techdocs.broadcom.com/us/en/vmware-cis/vsan/vsan/8-0/planning-and-deployment/designing-and-sizing-a-virtual-san-cluster/designing-the-virtual-san-network.html

This documentation specifies that "vSAN with RDMA does not support LACP, IP-hash-based NIC teaming, or route based on physical network adapter load. vSAN with RDMA does support NIC failover."