vSAN alarm of "invalid" for physical NIC's under "RDMA Mode" when vmnic down and up

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides information when physical NIC's have "invalid" on "RDMA Mode" under  vSAN cluster "Skyline health". 

The network card driver/firmware in use is : " nmlx5_core version 4.23.0.66 / Firmware version 26.39.1002 "

Symptoms:

• You navigate via vSphere UI,VSAN cluster object: Monitor -> vSAN-> Skyline Health and on RDMA Configuration Health you have "invalid" on "RDMA Mode" column.
• You bring vmnicX down and up with below the commands

            esxcli network nic down -n vmnicX

            esxcli nettwork nic up -n vmnicX

• You see the "Mode: 0"  with the following command  # esxcli network nic dcb status get -n vmnicX 

   Nic Name: vmnic1
  Mode: 0
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 8
         PFC Traffic Classes: 8
   PFC Enabled: true
   PFC Configuration: 0 0 0 1 0 0 0 0
   IEEE ETS Configuration:
         Willing Bit In ETS Config TLV: 1
         Supported Capacity: 8
         Credit Based Shaper ETS Algorithm Supported: 0x0
         TX Bandwidth Per TC: 13 13 13 13 12 12 12 12
         RX Bandwidth Per TC: 13 13 13 13 12 12 12 12

Environment

VMware vSphere ESXI  8.0.2.

Resolution

Resolution:

•  Upgrade nmlx5_core driver to version 4.23.6.2

Workaround:

•   Reboot the ESXI host

Additional Information

vSAN Health Service - Network Health - RDMA Configuration Health

https://knowledge.broadcom.com/external/article/326624/vsan-health-service-network-health-rdm.html

 

vSAN network alarm 'RDMA Configuration Health'" warning is seen in vSAN over RDMA cluster when using async Intel icen driver

https://knowledge.broadcom.com/external/article/317243/vsan-network-alarm-rdma-configuration-he.html