Configuring vSAN RDMA with Broadcom RDMA certified NICs
search cancel

Configuring vSAN RDMA with Broadcom RDMA certified NICs

book

Article ID: 394666

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

The intention of this KB is to be a central location for assisting with configuring Broadcom RDMA certified NICs for vSAN RDMA

If end to end vSAN RDMA deployment/configuration is not achieved on the switches/NICs RDMA will not function properly  

Environment

VMware vSAN (All Versions)

Resolution

  1. Ensure your Broadcom NICs are listed on the vSAN over RDMA compatibility guide
  2. Review KB Configuring RDMA for vSAN and Designing the vSAN Network "Using RDMA" section
  3. Enable RDMA Mode in your server BIOS, engage your server vendor if you need assistance gaining access/navigating the BIOS.
  4. Configure the NIC/switch for RDMA by following Configuring RDMA over Converged Ethernet (RoCE) on VMware
    Notes:
    For additional optimization see Advanced RDMA over Converged Ethernet (RoCE) Network Configuration
    Consult your switch vendor/documentation for enabling PFC/RDMA on the switch 
  5. Ensure to work with your switch vendor/networking team when configuring the switch for RDMA as Broadcom NICs are dependent on proper switch configuration for PFC to be enabled on the NICs.

Additional commands to gather/view NIC RDMA info

  • esxcli network nic dcb status get -n vmnicx - where x= the number of the vmnic
  • esxcfg-module -g bnxtnet

Additional Information

Powered by Wolken Software