RDMA adapter is not visible under physical adapters after installing RDMA driver
search cancel

RDMA adapter is not visible under physical adapters after installing RDMA driver

book

Article ID: 398868

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • After installing the RDMA driver on an ESXi host, the RDMA adapter is not visible under Networking ,Physical adapters

  • When running the command: esxcli rdma device list, no devices are listed

Environment

  • ESXi 8.x
  • ESX 9.x

Cause

Installing the drivers is not enough to enable RDMA on the adapter. The driver module must explicitly be initialise to switch on RDMA functionality.

Resolution

NOTE: Refer to vendor-specific documentation for precise configuration instructions and also Configuring RDMA for vSAN

Below is an example on how to enable RDMA for Intel Adapter on the ESXi host:

Step 1: SSH to ESXi Host

  • Connect to the ESXi host using SSH with root credentials.

Step 2: Check if RDMA Driver is Loaded

  • Run
    esxcli
    rdma device list

    If no devices are listed, the RDMA driver might not be loaded

Step 3: Verify and Load RDMA Driver Module

  • List loaded kernel modules:

    vmkload_mod -l | grep -i irdma
  • If not loaded, manually load the RDMA driver:

    vmkload_mod irdma
  • Confirm it loaded:

    vmkload_mod -s irdma

Step 4: Enable RDMA and RoCE on the Adapters

    For Intel NICs (icen driver):

  • Enable RDMA on ports 0 and 1:

    esxcli system module parameters set -m icen -p "RDMA=1,1"
  • Enable RoCE on irdman module:

    esxcli system module parameters set -m irdman -p "ROCE=1,1"

Details:

  • icen → Intel Ethernet devices (e.g., E810 series).

  • RDMA=1,1 → Enables RDMA on NIC port0 and port1.

  • irdman → Intel RDMA management driver.

  • ROCE=1,1 → Enables RoCE (RDMA over Converged Ethernet) on two interfaces.

 For ESXi Hosts with 4 RDMA NICs:

  • Enable RDMA and RoCE on all four ports:

     
    esxcli system module parameters set -m icen -p "RDMA=1,1,1,1"
    esxcli system module parameters set -m irdman -p "ROCE=1,1,1,1"

Note:
Each number represents a physical port (port0, port1, port2, port3).
1 = enable; 0 = disable.

To Verify Current Settings:

esxcli system module parameters list -m icen
esxcli system module parameters list -m irdman


Step 5: Apply Specific Port Configurations

Scenario Command
Enable RDMA/RoCE on vmnic0 & vmnic1; disable on vmnic2 & vmnic3 esxcli system module parameters set -m icen -p "RDMA=1,1,0,0"
esxcli system module parameters set -m irdman -p "ROCE=1,1,0,0"
Enable RDMA/RoCE on vmnic2 & vmnic3; disable on vmnic0 & vmnic1 esxcli system module parameters set -m icen -p "RDMA=0,0,1,1"
esxcli system module parameters set -m irdman -p "ROCE=0,0,1,1"
Disable RDMA/RoCE on all ports esxcli system module parameters set -m icen -p "RDMA=0,0,0,0"
esxcli system module parameters set -m irdman -p "ROCE=0,0,0,0"

Step 6: Reboot the Host

After applying the configuration changes, you must reboot the ESXi host for the settings to take effect.

Additional Information

  • If RDMA driver are installed and the modules are loaded the RDMA driver will be visible under ESXi-->Configure--->Networking tab

  • For Broadcom RDMA Nics, perform the below steps to enable rdma

    esxcli system module parameters set -m bnxtnet -p disable_roce=0
    esxcli system module parameters set -m bnxtroce -p disable_rocev2=0  (For ROCEv2 mode)

    Notes:
    ESX host reboot is required after enabling the modules.
    Ensure in the BIOS of the host, on the Device Level Configuration page, (NIC+RDMA mode) is Enabled for the RDMA NICs.