Delay in booting ESXi host: Stuck at vmkusb loaded successfully. activating: vmkdevmgr
search cancel

Delay in booting ESXi host: Stuck at vmkusb loaded successfully. activating: vmkdevmgr

book

Article ID: 403606

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

ESXi host is taking too long and gets stuck at:

vmkusb loaded successfully. activating: vmkdevmgr

vmkernel.log

  • These logs indicate repeated failures in initializing the Mellanox (nmlx5) NIC driver:
  • The ESXi kernel attempted to initialize the nmlx5 network adapter, but it failed due to the NIC's firmware being stuck in an unresponsive state. 
  • These messages suggest either a firmware hang, a driver/firmware version mismatch, or faulty hardware.
  • The driver attempts to recover via nmlx5_core_HealthDeviceRestartLocked, but that fails too—implying no recovery possible during boot.


[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2098563)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_AttachLocked - (nmlx5_core_main.c:3242) nmlx5_InitOne failed: Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2098563)DMA: 767: DMA Engine 'nmlx5_DxDmaEngineStreaming' destroyed.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2098563)DMA: 767: DMA Engine 'nmlx5_DxDmaEngineCoherent' destroyed.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2098563)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_AttachLocked - (nmlx5_core_main.c:3309) done  status: Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu36:2098563)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_HealthDeviceRestartLocked - (nmlx5_core_main.c:4823) nmlx5_core_AttachLocked failed: Retry the operation


[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu12:2098564)WARNING: <NMLX_ERR> 0000:b5:00.2: Firmware over 120000 MS in initializing state,stopping prematurely: Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)<NMLX_INF> Device internal error state is set
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu12:2098564)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_AttachLocked - (nmlx5_core_main.c:3242) nmlx5_InitOne failed: Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)DMA: 767: DMA Engine 'nmlx5_DxDmaEngineStreaming' destroyed.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)DMA: 767: DMA Engine 'nmlx5_DxDmaEngineCoherent' destroyed.
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_AttachLocked - (nmlx5_core_main.c:3309) done  status: Retry the operation
[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu12:2098564)<NMLX_ERR> nmlx5_cc: core: nmlx5_core_HealthDeviceRestartLocked - (nmlx5_core_main.c:4823) nmlx5_core_AttachLocked failed: Retry the operation

zcat vmkernel.0.gz | grep "nmlx5_InitOne failed" -wc

1268

  • This shows that the error nmlx5_InitOne failed occurred 1,268 times. 
  • Such a high count typically points to:
  • Repeated probing attempts during boot or polling loops
  • The volume of these failures strongly indicates a critical and unresolved issue at the driver or firmware level.

vmkwarning.log

  • These logs show that the NIC firmware did not complete its initialization even after 2 minutes (120,000ms).
  • VMware then forcefully stops the firmware initialization process. The repeated countdown logs (timeout halt in 100s, 80s, 59s...) show the driver polling for firmware readiness, which ultimately times out.
  • This indicates a firmware or PCI bus issue. 

[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_ERR> 0000:b5:00.2: Firmware over 120000 MS in initializing state,stopping prematurely: Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 100s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 79s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 100s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 59s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 80s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 39s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 59s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 19s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu21:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 39s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_ERR> 0000:0d:00.2: Firmware over 120000 MS in initializing state,stopping prematurely: Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu23:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 19s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu1:2098564)WARNING: <NMLX_ERR> 0000:b5:00.2: Firmware over 120000 MS in initializing state,stopping prematurely: Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu1:2098564)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 100s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 79s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu23:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 100s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 59s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu23:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 79s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 39s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu23:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 59s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_WRN> 0000:0d:00.2: Waiting for FW initialization, timeout halt in 19s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu9:2098564)WARNING: <NMLX_WRN> 0000:b5:00.2: Waiting for FW initialization, timeout halt in 39s
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_ERR> 0000:0d:00.2: Firmware over 120000 MS in initializing state,stopping prematurely: Retry the operation
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu36:2098563)WARNING: <NMLX_ERR> nmlx5_DevInit failed Retry the operation

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

The nmlx5_core driver is failing firmware initialization during early boot.

Resolution

Check and investigate with hardware vendor as to why the nmlx driver is reporting such failures. 

Powered by Wolken Software