Qlogic FastLinq QL41xxxx 10/25/40/50 GBE ethernet adapters go down every time the Esxi host reboots

Products

• VMware vSphere ESXi

Issue/Introduction

To ensure the NIC's stay up even after the host reboots for Qlogic FastLinq QL41xxxx  10/25/40/50 GBE ethernet adapters

Symptoms:

• Qlogic FastLinq QL41xxxx 10/25/40/50 GBE ethernet adapters go down everytime the Esxi host reboots.
• After the host reboot, esxcfg-nics -l command shows all VMNICs being down
• We have to manually run the command esxcli network nic up  -n vmnicX to get the network adapters up and online.
• The network cards are running on driver qedentv
• The vmkernel logs show that the driver qedentv successfully loaded

2019-03-27T00:04:25.393Z cpu7:66125)Loading module qedentv ...
2019-03-27T00:04:25.408Z cpu7:66125)QLogic FastLinQ QL45604 10/25/40/50/100 Gigabit Ethernet Driver, qedentv 3.0.7.5
2019-03-27T00:04:25.410Z cpu7:66125)Device: 191: Registered driver 'qedentv' from 18
2019-03-27T00:04:25.410Z cpu7:66125)[init_module:197(qedentv)]Initialization of qedentv driver successful

• However, after few seconds, the log show the vmnic's going down with below errors

2019-03-27T00:04:28.246Z cpu0:66236)qedentv: OSAL_LINK_UPDATE:994: NIC Link is down. vmnic0
2019-03-27T00:04:28.248Z cpu9:66234)[ecore_int_attentions:1172(vmnic1-0)]MFW indication via attention
2019-03-27T00:04:28.248Z cpu9:66234)[OSAL_LINK_UPDATE:981(vmnic1)]Duplex:1, Link_up:1, Speed:10000Mbps, udev->link_state:0
2019-03-27T00:04:28.544Z cpu6:66135)WARNING: qedentv: qedentv_advert_link_change:5182: vmk_UplinkUpdateLinkState failed: 0xbad0030, uplink: 0x4304d0e80c80
2019-03-27T00:04:28.544Z cpu9:66211)[ecore_int_attentions:1172(vmnic1-0)]MFW indication via attention

2019-03-27T00:04:28.571Z cpu9:66211)WARNING: [qedentv_txq_quiesce:392(vmnic1)]Either ctx:0 is freed or firmware is resetting.
2019-03-27T00:04:28.571Z cpu9:66211)WARNING: [qedentv_txq_quiesce:392(vmnic1)]Either ctx:1 is freed or firmware is resetting.
2019-03-27T00:04:28.571Z cpu9:66211)WARNING: [qedentv_txq_quiesce:392(vmnic1)]Either ctx:2 is freed or firmware is resetting.

• vobd.log has the following messages after the host boot up.

2019-03-27T00:10:19.021Z: [UserLevelCorrelator] 391844401us: [esx.audit.host.boot] Host has booted.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.flapping); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.flapping); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.flapping); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.flapping); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.flapping); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.clear.net.vmnic.linkstate.up); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.down); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.clear.net.vmnic.linkstate.up); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.down); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.clear.net.vmnic.linkstate.up); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.down); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.clear.net.vmnic.linkstate.up); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.problem.net.vmnic.linkstate.down); 3 failures so far.
2019-03-27T00:10:28.002Z: Failed to send event (esx.clear.net.vmnic.linkstate.up); 3 failures so far.

• Once we run the command esxcli network nic up -n vmnicX, to manually get the vmnic up, the network connectivity gets restored 

2019-03-27T00:36:54.973Z: [netCorrelator] 1987751645us: [vob.net.vmnic.linkstate.up] vmnic vmnic0 linkstate up
2019-03-27T00:36:56.001Z: [netCorrelator] 1988824482us: [esx.clear.net.connectivity.restored] Network connectivity restored on virtual switch "vSwitch0", portgroups: "vMotion", "Management Network". Physical NIC vmnic0 is up

Cause

Rebooting the esxi host causes the Ethernet adapters to go down every time and we had to run the command esxcli network nic up -n vmnicX to get the interface up. This behavior is observed on the qedentv drivers running with lower qedentv driver < 3.0.x

Resolution

To solve the issue schedule a maintenance activity and ensure to upgrade the firmware of the NIC adapters and driver version to latest version. Please refer the VMware HCL link to know the appropriate driver's and firmware's required for qedentv driver

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=45643&deviceCategory=io&details=1&partner=73&VID=1077&DID=8070&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

After updating the driver to 3.10.23.1 and firmware to 8.37, please reboot the host.

Additional Information

Impact/Risks:
Make sure the host is placed into maintenance mode and you do the firmware upgrade first before upgrading the driver.