Unable to see NIC device after enabling ROCE for Broadcom bnxtnet driver

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• In some platform with large pcpu numbers(e.g. 128), after enabling ROCE with "disable_roce=0" param for Broadcom bnxtnet driver and host reboot, user may not see the nic devices.
• The kernel log shows the following error during device attach:

YYYY-MM-DDTHH:MM:SS.060Z cpu93:2098540)MSI: 592: number of MSIX interrupts required: 75 > table len: 74
YYYY-MM-DDTHH:MM:SS.060Z cpu93:2098540)IntrCookie: 1564: Unable to allocate 75 cookies: Bad parameter
YYYY-MM-DDTHH:MM:SS.060Z cpu93:2098540)WARNING: VMK_PCI: 746: device 0000:19:00.0 failed to allocate 75 MSIX interrupts
YYYY-MM-DDTHH:MM:SS.060Z cpu93:2098540)WARNING: bnxtnet: bnxtnet_alloc_intr_resources:974: [0000:19:00.0 : 0x430d86e14050] Failed to allocate interrupt cookies (Bad parameter)
YYYY-MM-DDTHH:MM:SS.060Z cpu93:2098540)WARNING: bnxtnet: bnxtnet_set_int_mode:1185: [0000:19:00.0 : 0x430d86e14050] Failed to alloc interrupt resource in MSIX mode 
YYYY-MM-DDTHH:MM:SS.241Z cpu54:2098653)WARNING: bnxtnet: bnxtnet_attach_device:218: [0000:19:00.0 : 0x430d86e14050] failed to find cumulus device (status: Bad parameter)

Environment

VMware vSphere ESXi 7.0.0

Cause

• The cause of the issue has been linked to insufficient MISX interrupt number which is set in the BIOS - Device Setting.

Resolution

• To resolve the issue, increase the MISX interrupt number to a value large than the PCPU number in the "BIOS Setup- Device Setting page".
• Ensure drivers & firmware version numbers of the vmnics should start with the same octet as per below example:
   
  Drivers Version: 229.X.X.X
   Firmware Version: 229.X.X.X