RDMA on vSAN NIC is enabled, but not working

Products

• VMware vSAN
• VMware vSAN 8.x

Issue/Introduction

Symptoms:

RDMA is not working on the vSAN NIC despite having the following settings

• RDMA is enabled on NIC used for vSAN I/O.
• The physical network switches have been configured for PFC (Priority Flow Control) and DCBx (Data Center Bridging Exchange protocol).

Environment

• VMware vSAN 8.x

Cause

• The physical network adapter and/or related driver version used for vSAN I/O do not support the Data Center Bridging (DCB) feature.
  
  

Verification:

• When you run the following command from an ssh session on the host:

esxcli network nic dcb status get -n vmnicX

...you see the following error:

DCB not supported for NIC vmnicX: Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Not supported: VSI node (721:VSI_NODE_net_pNics_dcbx_dcbMode)

Resolution

• You will need to install a RDMA capable NIC that supports both PFC and DCB.
• Please contact your hardware vendor to verify the NIC does support the PFC and DCB features.
• Verify you have the correct NIC driver version installed. Please consult the Broadcom Compatibility Guide for NIC models and related driver versions that have been certified for support of vSAN over RDMA: Broadcom Compatibility Guide (vSAN over RDMA)

Additional Information

• For more information on configuring RMDA with vSAN, see:
  • KB: Configuring RDMA for vSAN
  • TechDocs: Configuring RoCE on VMware