vSAN Warning: "RDMA Configuration Health" - Priority Flow Control Not Enabled Due to DCB Mode Mismatch

Products

• VMware vSAN 8.x

Issue/Introduction

Symptoms

• vSAN cluster deployed with vSAN Express Storage Architecture (ESA) using RDMA transport.
• In the vSphere UI under vSAN > Skyline Health, a warning is displayed under “RDMA Configuration Health” for each vmnic used for vSAN over RDMA.

• The warning indicates Priority Flow Control (PFC) is not enabled.
• Example output from an ESXi host:

[root@ESXi:~] esxcli network nic dcb status get -n vmnic#
   Nic Name: vmnic#
   Mode: 3 - IEEE Mode
   Enabled: true
   Capabilities:
         Priority Group: true
         Priority Flow Control: true
         PG Traffic Classes: 8
         PFC Traffic Classes: 8
   PFC Enabled: false
   PFC Configuration: 0 0 0 0 0 0 0 0

Environment

• VMware vSAN 8.x

Cause

This issue occurs due to a Data Center Bridging (DCB) mode mismatch between the ESXi host and the connected physical switch.

• The ESXi host is configured to use DCB Mode 3 (IEEE).

• The physical switch is configured to use DCB Mode 2 (CEE).

As a result of this configuration mismatch, the ESXi host is unable to properly receive the DCB settings from the switch, which leads to Priority Flow Control (PFC) being disabled on the host.

Resolution

To resolve the issue, ensure that the Data Center Bridging (DCB) mode configuration is consistent between the ESXi host and the physical switch:

1. Engage the network switch vendor to review and validate the current switch configuration.

2. Align DCB modes between the ESXi host and the switch:

   • If the ESXi host is configured with DCB Mode 3 (IEEE), configure the switch to also use IEEE mode.

   • Alternatively, if the environment requires the switch to operate in CEE mode (DCB Mode 2), reconfigure the ESXi host to match accordingly.

3. Verify NIC compatibility:

   • Ensure that the network interface card (NIC) hardware and driver versions in use on the ESXi host support Priority Flow Control (PFC).

   • Refer to the Broadcom Compatibility Guide to confirm supported driver and firmware combinations.

After making the necessary configuration changes, verify that PFC is enabled and functioning correctly on the ESXi host.

Additional Information

For more details please refer -

• vSAN Health Service - Network Health - RDMA Configuration Health
• Configuring Lossless Ethernet for NVMe over RDMA