Step-by-Step Guide to Diagnosing Network Driver/Firmware Issues on ESXi Hosts with Log Files

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts can experience network problems due to mismatched drivers and firmware. This article explains how to diagnose these issues using log files

Environment

 

• VMware ESXi 7.x
• VMware ESXi 8.x

Cause

Mismatches between network adapter drivers and firmware versions can lead to communication disruptions, corrupted data exchanges, and overall network instability. These inconsistencies can manifest in various ways, including vMotion failures, network classification problems, and storage path issues. vMotion, a key technology for live migration of virtual machines, might fail due to incompatibility between the driver and firmware, while network classification problems can hinder proper network traffic prioritization and routing. Storage path issues can result in disrupted data access and slow performance, impacting critical operations

Resolution

While the bnxtnet driver is used as an example throughout this guide, the principles apply to any network driver from any manufacturer.


To diagnose potential network driver/firmware issues, follow these steps to analyze the ESXi host's vmkernel.log file:

1. Access the ESXi host's vmkernel.log file.
   a. Use the ESXi Shell or SSH into the ESXi host.
   b. Navigate to the `/var/log/` directory.
   c. Open the vmkernel.log file using a text editor like vi or nano.

2. Look for direct warnings about driver-firmware communication failures:
   
Example from vmkernel.log. 

   WARNING: bnxtnet: bnxtnet_check_link:3185: [vmnic4 : 0x400000000000] Unable to receive DCB data from firmware

Above entry indicates a potential problem between the bnxtnet driver and the firmware.

3. Identify messages showing invalid or unexpected data received by the driver:
   
Example from vmkernel.log:

   WARNING: bnxtnet: bnxtnet_dcb_get_setting:1653: [vmnic5 : 0x400000000000] invalid dcbx_cap 0

Above entry suggests the driver is receiving invalid Data Center Bridging (DCB) capability data.

4. Check for firmware trace dumps:
   
Example from vmkernel.log:

   bnxtnet: dev_dump_fw_trace:153: [vmnic5 : 0x400000000000] Dumping FW Trace:

Above entry stating Firmware trace dumps often indicate serious firmware-related problems.

5. Note any cascading effects such as network classification failures:
   
Example from vmkernel.log:

   NetX Classifier:  classifier port not found by vifId 00000000-0000-0000-0000-000000000000
   NetX IOCTL:  Failed to get flow prgramming table for vif 00000000-0000-0000-0000-000000000000: Failure

Above entry can be downstream effects of driver-firmware problems.

6. Collect adapter information for troubleshooting:
   
Example from vmkernel.log (the misspelling of "Adapter" is seen this way in logs):

   Adpater's NIC Info:-
   Driver bp[0x400000000000] BDF[0000:00:00.0] with Chip ID=0x0000 Rev/Metal:0/0

This information is crucial for identifying the specific hardware and firmware version in use.

7. Use esxcli commands to gather detailed information about network adapters and drivers:
   a. SSH into the ESXi host or use the ESXi Shell.
   b. Run the following commands to obtain detailed information:

   - List all network adapters:

    esxcli network nic list

   - Get detailed information about a specific network adapter (replace vmnic0 with the adapter you're troubleshooting):

     esxcli network nic get -n vmnic0

   - View driver information for a specific network adapter:

     esxcli network nic driver info -n vmnicX  <== in place of 'X; substitute different vmnic numbers, vmnic0, vmnic1, etc

   - List all loaded drivers:

     esxcli system module list | grep -i net

   - Get detailed information about a specific driver (replace bnxtnet with the driver you're investigating):

     esxcli system module get -m bnxtnet

   - View the driver parameters for a specific network adapter:

     esxcli system module parameters list -m bnxtnet

   c. Analyze the output of these commands to gather information about:
      - Driver version
      - Firmware version
      - Driver parameters
      - Link status
      - Supported features

   d. Compare the gathered information with the vendor's specifications and recommended versions to identify any mismatches or outdated components.

8. After identifying these issues, check for driver and firmware version compatibility.
   a. Compare the installed versions with the latest recommended versions from your hardware vendor.
   b. If mismatches are found, plan to update the driver and firmware to compatible versions.

Additional Information

- For specific update procedures, refer to your hardware vendor's documentation.
Broadcom Compatibility Guide