Network adapters are failing to come online on ESX host

Products

• VMware vSphere ESX 8.x

Issue/Introduction

 Network adapters are failing to come online on ESX host, these network cards report MAC address of 00:00:00:00:00:00 in vCenter.

 

Environment

vSphere ESX 

Cause

Following is the output of command 'esxcfg-nics -l' 

 

 vmnic   PCI bus address  link  speed   duplex  MTU   driver      driver version  firmware version  MAC address        VID   DID   SVID  SDID  name
  -----   ---------------  ----  -----   ------  ---   ------      --------------  ----------------  -----------        ---   ---   ----  ----  -----------------------------------
  vmnic0  0000:ab:00.0     Up    100000  Full    9000  nmlx5_core  4.23.6.5        22.39.1002        ##:##:##:##:##:##  15b3  101d  15b3  0016  Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AN-CDA)
  vmnic1  0000:ab:00.1     Up    100000  Full    9000  nmlx5_core  4.23.6.5        22.39.1002        ##:##:##:##:##:##  15b3  101d  15b3  0016  Mellanox Technologies ConnectX-6 Dx EN NIC; 100GbE; dual-port QSFP56; PCIe4.0 x16; (MCX623106AN-CDA)
  vmnic4  0000:17:00.0     Up    25000   Full    9000  nmlx5_core  4.23.6.5        26.35.2000        ##:##:##:##:##:##  15b3  101f  15d9  1c1b  Mellanox Technologies MT2894 Family [ConnectX-6 Lx]
  vmnic5  0000:17:00.1     Up    25000   Full    9000  nmlx5_core  4.23.6.5        26.35.2000        ##:##:##:##:##:##  15b3  101f  15d9  1c1b  Mellanox Technologies MT2894 Family [ConnectX-6 Lx]

 
Firmware error on ESX startup logged in /var/log/boot

Example from the boot log

2025-09-09T15:11:01.990Z cpu14:2098176)<NMLX_ERR> nmlx5_core: 0000:3d:00.0: Health: Miss counters detected
2025-09-09T15:11:01.990Z cpu14:2098176)<NMLX_INF> Device internal error state is set
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertVar[0] 0x00000088
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertVar[1] 0x00000000
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertVar[2] 0x00000000
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertVar[3] 0xffffffff
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertVar[4] 0x00000000
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertExitPtr 0x2089eb24
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> assertCallra 0x20897990
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> firmwareVersion 0x182307d0
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> hwId 0x00010214
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> iriscIndex 5
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> synd 0x1: firmware internal error
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> extSynd 0x8574
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> driver 4.23.6.5
2025-09-09T15:11:01.993Z cpu14:2098176)<NMLX_INF> nmlx5_core: 0000:3d:00.0: Health: thread is stopped 0x43247b64ee08
2025-09-09T15:11:02.070Z cpu61:2098210)<NMLX_ERR> nmlx5_core: 0000:3d:00.1: Health: Miss counters detected
2025-09-09T15:11:02.070Z cpu61:2098210)<NMLX_INF> Device internal error state is set
2025-09-09T15:11:02.073Z cpu61:2098210)<NMLX_INF> assertVar[0] 0x00000088
2025-09-09T15:11:02.073Z cpu61:2098210)<NMLX_INF> assertVar[1] 0x00000000
2025-09-09T15:11:02.073Z cpu61:2098210)<NMLX_INF> assertVar[2] 0x00000000
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> assertVar[3] 0xffffffff
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> assertVar[4] 0x00000000
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> assertExitPtr 0x2089eb24
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> assertCallra 0x20897990
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> firmwareVersion 0x182307d0
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> hwId 0x00010214
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> iriscIndex 5
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> synd 0x1: firmware internal error
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> extSynd 0x8574
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> driver 4.23.6.5
2025-09-09T15:11:02.074Z cpu61:2098210)<NMLX_INF> nmlx5_core: 0000:3d:00.1: Health: thread is stopped 0x43247b64f748

 

Since firmware is crashing on boot, the NICs are not fully coming up in ESXi, 

Resolution

The issue is due to firmware version and firmware require upgrade.

Reference: https://compatibilityguide.broadcom.com/detail?program=io&productId=50286