NICs unexpectedly go through multiple link down/up events on HPE servers when iLO is configured for automatic firmware updates

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x
• VMware vSphere ESXi 8.0

Issue/Introduction

A VCF administrator will observe NICs go thru multiple link down/up events: 

2026-02-10T08:30:43.149Z In(14) vobd[2097764]:  [netCorrelator] 1705146023930us: [vob.net.vmnic.linkstate.down] vmnic vmnic4 linkstate down
2026-02-10T08:30:43.149Z In(14) vobd[2097764]:  [netCorrelator] 1705146024022us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
2026-02-10T08:30:43.149Z In(14) vobd[2097764]:  [netCorrelator] 1705146024286us: [vob.net.vmnic.linkstate.down] vmnic vmnic6 linkstate down
2026-02-10T08:30:43.149Z In(14) vobd[2097764]:  [netCorrelator] 1705146024508us: [vob.net.vmnic.linkstate.down] vmnic vmnic7 linkstate down
2026-02-10T08:30:45.282Z In(14) vobd[2097764]:  [netCorrelator] 1705148158531us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down
2026-02-10T08:30:45.282Z In(14) vobd[2097764]:  [netCorrelator] 1705148158557us: [vob.net.vmnic.linkstate.down] vmnic vmnic0 linkstate down
2026-02-10T08:30:45.282Z In(14) vobd[2097764]:  [netCorrelator] 1705148158724us: [vob.net.vmnic.linkstate.down] vmnic vmnic3 linkstate down
2026-02-10T08:30:45.282Z In(14) vobd[2097764]:  [netCorrelator] 1705148158885us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
2026-02-10T08:30:55.435Z In(14) vobd[2097764]:  [netCorrelator] 1705158311185us: [vob.net.vmnic.linkstate.down] vmnic vmnic4 linkstate down
2026-02-10T08:30:55.511Z In(14) vobd[2097764]:  [netCorrelator] 1705158387782us: [vob.net.vmnic.linkstate.down] vmnic vmnic7 linkstate down
2026-02-10T08:30:55.545Z In(14) vobd[2097764]:  [netCorrelator] 1705158421586us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
2026-02-10T08:30:55.580Z In(14) vobd[2097764]:  [netCorrelator] 1705158456916us: [vob.net.vmnic.linkstate.down] vmnic vmnic6 linkstate down
2026-02-10T08:30:57.980Z In(14) vobd[2097764]:  [netCorrelator] 1705160856442us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
2026-02-10T08:30:58.049Z In(14) vobd[2097764]:  [netCorrelator] 1705160925950us: [vob.net.vmnic.linkstate.down] vmnic vmnic0 linkstate down
2026-02-10T08:30:58.245Z In(14) vobd[2097764]:  [netCorrelator] 1705161122350us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down
2026-02-10T08:30:58.253Z In(14) vobd[2097764]:  [netCorrelator] 1705161129920us: [vob.net.vmnic.linkstate.down] vmnic vmnic3 linkstate down
2026-02-10T08:31:09.594Z In(14) vobd[2097764]:  [netCorrelator] 1705172469910us: [vob.net.vmnic.linkstate.up] vmnic vmnic5 linkstate up
2026-02-10T08:31:09.780Z In(14) vobd[2097764]:  [netCorrelator] 1705172656379us: [vob.net.vmnic.linkstate.up] vmnic vmnic4 linkstate up
2026-02-10T08:31:10.190Z In(14) vobd[2097764]:  [netCorrelator] 1705173066217us: [vob.net.vmnic.linkstate.up] vmnic vmnic7 linkstate up
2026-02-10T08:31:10.190Z In(14) vobd[2097764]:  [netCorrelator] 1705173067105us: [vob.net.vmnic.linkstate.up] vmnic vmnic6 linkstate up
2026-02-10T08:31:11.820Z In(14) vobd[2097764]:  [netCorrelator] 1705174696987us: [vob.net.vmnic.linkstate.up] vmnic vmnic0 linkstate up
2026-02-10T08:31:11.920Z In(14) vobd[2097764]:  [netCorrelator] 1705174796341us: [vob.net.vmnic.linkstate.up] vmnic vmnic3 linkstate up
2026-02-10T08:31:12.024Z In(14) vobd[2097764]:  [netCorrelator] 1705174901109us: [vob.net.vmnic.linkstate.up] vmnic vmnic1 linkstate up
2026-02-10T08:31:12.025Z In(14) vobd[2097764]:  [netCorrelator] 1705174901188us: [vob.net.vmnic.linkstate.up] vmnic vmnic2 linkstate up
2026-02-10T08:32:38.700Z In(14) vobd[2097764]:  [netCorrelator] 1705261576901us: [vob.net.vmnic.linkstate.down] vmnic vmnic4 linkstate down
2026-02-10T08:32:38.701Z In(14) vobd[2097764]:  [netCorrelator] 1705261577095us: [vob.net.vmnic.linkstate.down] vmnic vmnic7 linkstate down
2026-02-10T08:32:38.701Z In(14) vobd[2097764]:  [netCorrelator] 1705261577318us: [vob.net.vmnic.linkstate.down] vmnic vmnic6 linkstate down
2026-02-10T08:32:38.701Z In(14) vobd[2097764]:  [netCorrelator] 1705261577507us: [vob.net.vmnic.linkstate.down] vmnic vmnic5 linkstate down
2026-02-10T08:32:46.257Z In(14) vobd[2097764]:  [netCorrelator] 1705269133161us: [vob.net.vmnic.linkstate.down] vmnic vmnic0 linkstate down
2026-02-10T08:32:46.257Z In(14) vobd[2097764]:  [netCorrelator] 1705269133222us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down
2026-02-10T08:32:46.257Z In(14) vobd[2097764]:  [netCorrelator] 1705269133478us: [vob.net.vmnic.linkstate.down] vmnic vmnic3 linkstate down
2026-02-10T08:32:46.257Z In(14) vobd[2097764]:  [netCorrelator] 1705269133632us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
2026-02-10T08:32:58.354Z In(14) vobd[2097764]:  [netCorrelator] 1705281230159us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
2026-02-10T08:32:58.442Z In(14) vobd[2097764]:  [netCorrelator] 1705281319291us: [vob.net.vmnic.linkstate.down] vmnic vmnic3 linkstate down
2026-02-10T08:32:58.551Z In(14) vobd[2097764]:  [netCorrelator] 1705281427630us: [vob.net.vmnic.linkstate.down] vmnic vmnic2 linkstate down
2026-02-10T08:32:58.610Z In(14) vobd[2097764]:  [netCorrelator] 1705281487273us: [vob.net.vmnic.linkstate.down] vmnic vmnic0 linkstate down
2026-02-10T08:33:02.935Z In(14) vobd[2097764]:  [netCorrelator] 1705285811483us: [vob.net.vmnic.linkstate.up] vmnic vmnic0 linkstate up
2026-02-10T08:33:02.935Z In(14) vobd[2097764]:  [netCorrelator] 1705285811620us: [vob.net.vmnic.linkstate.up] vmnic vmnic1 linkstate up
2026-02-10T08:33:02.938Z In(14) vobd[2097764]:  [netCorrelator] 1705285814889us: [vob.net.vmnic.linkstate.up] vmnic vmnic2 linkstate up
2026-02-10T08:33:03.122Z In(14) vobd[2097764]:  [netCorrelator] 1705285998960us: [vob.net.vmnic.linkstate.up] vmnic vmnic3 linkstate up

This could result in the NICs eventually going non-functional until the host is rebooted when the firmware of the NIC performs a rollback, as reported by the driver:

2026-02-10T08:32:58.066Z Wa(180) vmkwarning: cpu52:2097954)WARNING: icen: 12:00.1: Firmware rollback mode detected. Current version is NVM: 4.50 0x8001d8b5 1.3755.0, FW: 7.5.
2026-02-10T08:32:58.066Z Wa(180) vmkwarning: cpu52:2097954)WARNING: icen: 12:00.1: Device may exhibit limited functionality. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware rollback mode.
2026-02-10T08:32:58.072Z Wa(180) vmkwarning: cpu52:2097954)WARNING: icen: 12:00.1: Get PHY capabilities failed status = -100, continuing anyway
2026-02-10T08:32:58.264Z In(182) vmkernel: cpu52:2097954)icen: icen_ValidatePkgState:114: 0000:12:00.1: The DDP package was successfully loaded: ICE OS Default Package version 1.3.39.1.
2026-02-10T08:32:58.264Z In(182) vmkernel: cpu52:2097954)icen: icen_SetPfDvm:1531: 0000:12:00.1: Network adapter is configured in Double VLAN Mode
2026-02-10T08:32:58.268Z In(182) vmkernel: cpu52:2097954)icen: icen_StartStopLLDP:9780: 0000:12:00.1: Firmware Link Layer Discovery Protocol (FW LLDP) agent is disabled
2026-02-10T08:32:58.353Z Wa(180) vmkwarning: cpu52:2097954)WARNING: icen: icen_GetLinkStatus:1268: 0000:12:00.1: Failed to get link state - status: ICE_ERR_AQ_ERROR
2026-02-10T08:32:58.360Z Wa(180) vmkwarning: cpu35:2098028)WARNING: icen: 12:00.3: Firmware rollback mode detected. Current version is NVM: 4.50 0x8001d8b5 1.3755.0, FW: 7.5.
2026-02-10T08:32:58.360Z Wa(180) vmkwarning: cpu35:2098028)WARNING: icen: 12:00.3: Device may exhibit limited functionality. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware rollback mode.
2026-02-10T08:32:58.365Z Wa(180) vmkwarning: cpu35:2098028)WARNING: icen: 12:00.3: Get PHY capabilities failed status = -100, continuing anyway
2026-02-10T08:32:58.366Z In(182) vmkernel: cpu35:2098028)icen: icen_ValidatePkgState:123: 0000:12:00.3: The DDP package was already loaded: ICE OS Default Package version 1.3.39.1.
2026-02-10T08:32:58.366Z In(182) vmkernel: cpu35:2098028)icen: icen_SetPfDvm:1531: 0000:12:00.3: Network adapter is configured in Double VLAN Mode
2026-02-10T08:32:58.371Z In(182) vmkernel: cpu35:2098028)icen: icen_StartStopLLDP:9780: 0000:12:00.3: Firmware Link Layer Discovery Protocol (FW LLDP) agent is disabled
2026-02-10T08:32:58.407Z In(182) vmkernel: cpu52:2097954)icen: indrv_updateSchedulerTree:496: UpdateSchedulerTree: 0x431e49801240 0x431e49801240 0x0 1
2026-02-10T08:32:58.470Z Wa(180) vmkwarning: cpu54:2097989)WARNING: icen: 12:00.2: Firmware rollback mode detected. Current version is NVM: 4.50 0x8001d8b5 1.3755.0, FW: 7.5.
2026-02-10T08:32:58.470Z Wa(180) vmkwarning: cpu54:2097989)WARNING: icen: 12:00.2: Device may exhibit limited functionality. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware rollback mode.
2026-02-10T08:32:58.475Z Wa(180) vmkwarning: cpu54:2097989)WARNING: icen: 12:00.2: Get PHY capabilities failed status = -100, continuing anyway
2026-02-10T08:32:58.475Z In(182) vmkernel: cpu54:2097989)icen: icen_ValidatePkgState:123: 0000:12:00.2: The DDP package was already loaded: ICE OS Default Package version 1.3.39.1.
2026-02-10T08:32:58.475Z In(182) vmkernel: cpu54:2097989)icen: icen_SetPfDvm:1531: 0000:12:00.2: Network adapter is configured in Double VLAN Mode
2026-02-10T08:32:58.479Z In(182) vmkernel: cpu54:2097989)icen: icen_StartStopLLDP:9780: 0000:12:00.2: Firmware Link Layer Discovery Protocol (FW LLDP) agent is disabled
2026-02-10T08:32:58.491Z In(182) vmkernel: cpu35:2098028)icen: indrv_updateSchedulerTree:496: UpdateSchedulerTree: 0x431ea4801240 0x431ea4801240 0x1 1
2026-02-10T08:32:58.520Z Wa(180) vmkwarning: cpu46:2097919)WARNING: icen: 12:00.0: Firmware rollback mode detected. Current version is NVM: 4.50 0x8001d8b5 1.3755.0, FW: 7.5.
2026-02-10T08:32:58.520Z Wa(180) vmkwarning: cpu46:2097919)WARNING: icen: 12:00.0: Device may exhibit limited functionality. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware rollback mode.
2026-02-10T08:32:58.524Z Wa(180) vmkwarning: cpu46:2097919)WARNING: icen: 12:00.0: Get PHY capabilities failed status = -100, continuing anyway
2026-02-10T08:32:58.525Z In(182) vmkernel: cpu46:2097919)icen: icen_ValidatePkgState:123: 0000:12:00.0: The DDP package was already loaded: ICE OS Default Package version 1.3.39.1.
2026-02-10T08:32:58.526Z In(182) vmkernel: cpu46:2097919)icen: icen_SetPfDvm:1531: 0000:12:00.0: Network adapter is configured in Double VLAN Mode
2026-02-10T08:32:58.536Z In(182) vmkernel: cpu46:2097919)icen: icen_StartStopLLDP:9780: 0000:12:00.0: Firmware Link Layer Discovery Protocol (FW LLDP) agent is disabled

Environment

ESXi (all versions)
HPE servers

Cause

If the iLO is configured to perform automatic firmware updates (which is enabled by default), then this could result in NIC firmware also being applied as part of that process. When the firmware is applied to the NICs, this will result in link down/up events as the new firmware is loaded. In some cases, the firmware may have to rollback afterwards if the firmware did not apply correct, which could lead to the NICs no longer functioning until the host is rebooted.

Resolution

This is avoidable by simply disabling automatic firmware updates in HPE's iLO interface. By doing this, administrators will maintain control of their servers and when they perform firmware updates. For more information, see HPE's official documentation: Automatic and on-demand iLO firmware updates