Warning: HWRM cmd error, cmd_type 0x90(HWRM_CFA_L2_FILTER_ALLOC) error 0x2(INVALID_PARAMS) seq <sequence number>

Products

• VMware vSphere ESXi

Issue/Introduction

• The ESXi host unexpectedly starts facing connectivity issues, resulting in it being reported as "Not Responding" or "Down" in vCenter.
• Virtual machines connected to the impacted host also report intermittent or complete connectivity loss.
• The port group where the impacted host and VMs are connected has multiple physical adapters (vmnic) configured in an active-active failover order.
• Reviewing the /var/log/vmkernel.log file, you will find Hardware Resource Management (HWRM) command errors indicating a failure to allocate L2 filters:

YYYY-MM-DDTHH:MM:SS.285Z Wa(180) vmkwarning: cpu46:2097517)WARNING: bnxtnet: hwrm_send_msg:441: [vmnic# : 0x431202e91680] HWRM cmd error, cmd_type 0x90(HWRM_CFA_L2_FILTER_ALLOC) error 0x2(INVALID_PARAMS) seq 17913
YYYY-MM-DDTHH:MM:SS.324Z In(182) vmkernel: cpu42:2097517)bnxtnet: dev_dump_fw_trace:154: [vmnic# : 0x431202e91680] Dumping FW Trace:
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel:
YYYY-MM-DDTHH:MM:SS.324Z In(182) vmkernel: cpu42:2097517)bnxtnet: dev_print_fw_data:124: [vmnic# : 0x431202e91680] == START OF TRACE ==
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel:
YYYY-MM-DDTHH:MM:SSZ In(182) vmkernel: 0.5:*** Start of Trace FW version 214.0.253.1 ***
...
YYYY-MM-DDTHH:MM:SS.325Z In(182) vmkernel: cpu42:2097517)bnxtnet: dev_print_fw_data:136: [vmnic# : 0x431202e91680] == END OF TRACE ==

NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi

Cause

This is caused by a firmware error of the physical network card.

Resolution

Recommended Action

1. Validate Physical Health: Engage your hardware vendor (e.g., Dell, HPE, Lenovo) to validate the physical health and integrity of the network adapters.
2. Audit Current Versions: Identify the current BIOS, NIC firmware, and driver versions using KB 323110: Determining Network/Storage firmware and driver version in ESXi.
3. Ensure Driver/Firmware Alignment:
   • Update the NIC drivers and firmware to the latest supported versions.
   • Verify that the driver and firmware are a certified matched pair according to the VMware Compatibility Guide for IO Devices.
4. Review Configuration: Confirm that network configuration settings (such as RSS or NetQueue settings) align with both vendor and VMware best practices.

Additional Information

For validating the driver/firmware version of the adapters, please refer article: Determining Network/Storage firmware and driver version in ESXi

Also, kindly refer the Broadcom compatibility guide: Broadcom Compatibility Guide for IO devices