VMs freeze or restart due to issues with QLogic FastLinQ Adapters on ESXi hosts.
Article ID: 448987
Updated On:
Products
Issue/Introduction
- Virtual machines experience functional disruption including, but not limited to:
- Blocked guest I/O and network processing, causing PVSCSI Reset.
- Abrupt restarts or reboots triggered by vSphere High Availability (HA).
- The vSphere Client may display "
VmRestartedByHAEvent" for multiple VMs. - The incident is often preceded by rapid shifting of receive rules or queue modifications on the physical NICs and unusual amount of destroyed and recreated storage target paths.
Environment
VMware ESXi 8.0.x
Drivers Affected:
- qedf version 2.74.10.0
- qedrntv version 3.71.80.0
- qedentv version 3.71.80.0
- qedrntv version 3.71.90.2
- qedentv version 3.71.90.2
- qedi version 2.74.10.0
Cause
This issue is caused by a known defect in the qedentv driver versions 3.71.80.0 and 3.71.90.2, and driver qedf version 2.74.10.0.
During rapid queue allocation or teardown, the driver can enter a spin-lock or infinite-wait condition. This results in 100% CPU utilization on the associated "EnsNetWorlds" CPU cores, preventing the host from processing storage heartbeats and I/O commands in a timely manner. Consequently, the host loses connectivity to its datastores, triggering a VMware HA failover.
- In
/var/run/log/vmkernel.log, the following pattern may be seen:
- Messages as the example below will appear multiple times when the physical network interfaces re-negotiated or re-initialized their RX/TX queues numerous times in a short period:
vmkernel: cpu80:16458725)Uplink: 559: vmnic#: Driver claims supporting 25 RX queues, and 25 queues are accepted.
- The host's QLogic FCoE/FC driver (qedf) rapidly destroys and recreates numerous times storage target paths across vmhbas as in the sample below:
qedf:vmhba64:qedfc_discover:2504:Info: found rport TGT ID=7qedf:vmhba64:qedfc_discover:2532:Info: creating path vmhba64:0:7:0qedf:vmhba64:qedfc_discover:2549:Info: Destroying path vmhba64:0:7:0qedf:vmhba65:qedfc_discover:2504:Info: found rport TGT ID=0qedf:vmhba65:qedfc_discover:2532:Info: creating path vmhba65:0:0:0qedf:vmhba65:qedfc_discover:2549:Info: Destroying path vmhba65:0:0:0
- ESXi host hits memory admission failure:
vmkernel: cpu57:16####27)Admission failure in path: host/user/vm.16####15:vmmanon.16####15vmkernel: cpu57:16####27)vmmanon.16####15 (11######79) requires 1812 KB, asked 1812 KB from vm.16####15 (11#####49) which has 452864 KB occupied and 1792 KB available.
- In
/var/run/log/vmkernel.log, multiple shared datastores may report heartbeat timeouts with messages similar to the samples:
WARNING: NSX BFD session "State Down" Flags: [Poll]
WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: NMP device "naa.###" state in doubt
vmfs.heartbeat.timedout
Resolution
To resolve this issue, the qedentv driver must be updated to a version containing the fix for the spin-lock bug.
Verify the current driver version: Run the following command on the impacted ESXi host:
esxcli software vib list | grep qedentv- Align Firmware and Driver: Engage your hardware vendor to download and install a certified
qedentvdriver and corresponding firmware where this known issue is fixed. - Check Hardware Compatibility: Before upgrading, consult the to ensure that your current NIC firmware is compatible with the target driver version.
Reboot: A host reboot is required for the new driver and firmware changes to take effect.
Additional Information
Feedback
