vCenter Server and ESXi hosts report "Network redundancy lost" on adapters using the qedentv driver

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

ESXi hosts experience intermittent network connectivity loss or "Network Redundancy Lost" alarms. This occurs on physical adapters using the qedentv driver, even when physical switch ports show no link-down events.

• vCenter reports "Network Redundancy Lost" for specific vmnics.
• Connectivity intermittently drops and may recover only after a host reboot.
• Host becomes temporarily unmanaged in vCenter during the event.
• NIC performs a "GRC Dump" to capture diagnostic data during a hardware failure.

vmkernel log entries indicate the driver is triggering firmware dumps due to hardware-level resets:

/var/log/vmkernel.log

Wa(180) vmkwarning: cpu61:#####)WARNING: [qedentv_send_mcp_raw_debug_data:87(vmnic#-#)]Failed to send raw data
In(182) vmkernel: cpu61:#####)[qedentv_fw_collect_dump:1959(vmnic#)]Collect FW dump invoked during uplink reset
In(182) vmkernel: cpu61:#####)[qedentv_fw_collect_dump:1962(vmnic#)]Uplink Firmware dump already present at 0x431739dfd8a0 for size in dwords) 1217831.
In(182) vmkernel: cpu61:#####)[qedentv_uplink_reset:2348(vmnic#)]Invoking fw_dump_Handler udev->uplink_fw_dump_ptr is at 0x431739dfd8a0 and udev->uplink_fw_dump_size is 4871324 and udev_add is 0x#####
In(182) vmkernel: cpu61:#####)[qedentv_uplink_reset:2352(vmnic#)]Run `esxcli network nic eeprom dump -l #### -o #### -n vmnic# > uplinkreset_debug_data_output.hex' to dump uplink reset debug data --------> -l and -o are length and offset. 

In(182) vmkernel: cpu14:#####)[qedentv_fw_collect_dump_size:1667(vmnic#-#)]Dbg bus dump Buffer size (in dwords) is 3177 Total Size 3181 
In(182) vmkernel: cpu14:#####)[qedentv_fw_collect_dump_size:1774(vmnic#-#)]fw_asserts buffer size (in dwords) is 187 Total size 36548 
In(182) vmkernel: cpu14:#####)[qedentv_fw_collect_dump_size:1861(vmnic#-#)]Link Dump size (in dwords) is 4096 Total size 619812 
In(182) vmkernel: cpu14:#####)[qedentv_fw_collect_dump_size:1890(vmnic#-#)]GRC Dump buffer size (in dwords) is 530724 Total size 1166673

Environment

VMware vSphere ESXi 

 

Cause

The network redundancy loss is caused by continuous hardware or firmware-level resets of the physical network interface cards (NICs), utilizing the qedentv driver.

Log events indicate the qedentv_uplink_reset function is being invoked and generating firmware dumps. This shows the network drops originating from physical adapter hardware/firmware faults rather than a vSphere networking configuration or software defect.

Resolution

The issue is primarily addressed by aligning firmware and driver versions with the hardware vendor's latest certified releases. Use the following workaround to stabilize the environment.

Identify and Update the driver

1. Run the following command to verify the current qedentv driver version: esxcli network nic get -n vmnicX
   Note: Replace vmnicX with the affected adapter name
2. Download and install the latest certified async driver from the Broadcom Compatibility Guide

Verify Firmware Alignment and Engage Hardware Vendor

1. Ensure the NIC firmware version is compatible with the installed driver using the Broadcom Compatibility Guide.
2. Firmware crashes and physical hardware resets are outside the scope of VMware by Broadcom software support. Open a support request with the host hardware vendor to investigate the root cause of the firmware resets. The vendor may need to engage the NIC manufacturer (Marvell/QLogic) to analyze the logs.
3. To provide diagnostic data to the vendor, run: esxcli network nic eeprom dump -l ##### -o ##### -n vmnicX > uplinkreset_debug_data_output.hex
   Note: Check the vmkernel.log for the exact -l (length) and -o (offset) numbers generated during the crash and substitute them for the ### placeholders

Workaround

If an immediate update is not possible, you can disable periodic statistics polling to reduce driver-to-firmware communication overhead:

1. Login to Host SSH with root credential
2. Run the command: esxcli system module parameters set -m qedentv -p "en_periodic_stats=0"
3. Reboot the ESXi host to apply the changes.
4. Verify the parameter configuration by running: esxcli system module parameters list -m qedentv | grep en_periodic_stats
   
   Example output: 
   [root@host fqdn:~] esxcli system module parameters list -m qedentv | grep en_periodic_stats
en_periodic_stats   uint           0      Enable/Disable capturing periodic stats. Enable = 1, Disable = 0. [Default: 1]