ESXI host reboot gets hung at "nenic loaded successfully / activating: vmkdevmgr"

Products

• VMware vSphere ESXi

Issue/Introduction

• When a warm reboot is initiated on the host, host fails to boot and hangs at "nenic loaded successfully / activating: vmkdevmgr)
• Issue noticed with Cisco Hardware

/var/run/log/vmkernel.log:

YYYY-MM-DDtHH:MM:SS cpu:####### )nvmerdma:### vmrdma#, failed to get DMA memory region: Timeout
YYYY-MM-DDtHH:MM:SS cpu:####### )nvmerdma:### vmrdma#, failed to get DMA memory region: Timeout
YYYY-MM-DDtHH:MM:SS cpu:####### )nvmerdma:### vmrdma#, failed to get DMA memory region: Timeout
YYYY-MM-DDtHH:MM:SS cpu:####### )nvmerdma:### vmrdma#, failed to get DMA memory region: Timeout

YYYY-MM-DDtHH:MM:SS cpu:####### )Device: ###: Device 0x####### scan from driver nvmerdma failed with bad0039
YYYY-MM-DDtHH:MM:SS cpu:####### )Device: ###: Device 0x#######  scan from driver nvmerdma failed with bad0039
YYYY-MM-DDtHH:MM:SS cpu:####### )Device: ###: Device 0x#######  scan from driver nvmerdma failed with bad0021
YYYY-MM-DDtHH:MM:SS cpu:####### )Device: ###: Device 0x#######  scan from driver nvmerdma failed with bad0021
YYYY-MM-DDtHH:MM:SS cpu:####### )Device: ###: Device 0x#######  scan from driver nvmerdma failed with bad0021

Environment

VMware vSphere ESXi 8.x

Cause

It is found to be a hardware issue and the VIC card(RDMA) will have to be replaced

Resolution

Please work with Hardware vendor to get the Virtual Interface card (VIC) replaced.

Once the driver is replaced, ensure the drivers and firmwares are compatibile as per VMwareByBroadcom Compatibility Guide