PSOD on Cisco servers, when NFNIC 5.0.0.46 is in use

Products

• VMware vSphere ESXi

Issue/Introduction

• PSOD occurs on Hosts when nfnic 5.0.0.46-1OEM.803.0.0.24022510 is in use
  
  
  
  VMware ESXi 8.0.3 [Releasebuild-24784735 x86_64]

#PF Exception 14 in world 2111325:fnic_world_f IP 0x42001657fccl addr 0x45b9422cd2d0
PTEs:0x3040071023:0x135a93063:0x0;

Module(s) involved in panic: [nfnic 5.0.0.46-10EM.803.0.0.24022510 (External)]
cr0=0x8001003d cr2=0x45b9422cd2d0 cr3=0x6c867000 cr4=0x14216c
FMS=06/55/7 uCode=0x5003801
frame=0x453928f9bdb0 ip=0x42001657fccl err=0x0 rflags=0x10283
rax=0xfffff6894d5a1557 rbx=0x4311d80013c0 rcx=0x35
rdx=0x763 rbp=0x4311e4a2b0f8 rsi=0x0
rdi=0x98360931a6c r8=0x4311e4a2b0f8 r9=0x7b8eed
r10=0x3fc00000 г11=0x420047401040 г12=0x405
r13=0x0 r14=0x4200165abc28 r15=0x45b9422cd280
*PCPU9:2111325/fnic_world_func-1-0x4311d8079ba
PCPU 0: SSSSSSSUSSUUVSSVVSSSVSSSSSVSSSVSSSUVUSSSUVSSVV
Code start: 0x420015400000 VMK uptine: 0:01:04:34.067
0x453928f9be70: [0x42001657fcc1]fnic_abort_cnd@(nfnic)#<None>+0x921 stack: 0x976b2e10792
0x453928f9bf00: [0x42001658041b]fnic_virt_reset_abort@(nfnic)#<None>+0x1d0 stack: 0x4311d8079ba0
0x453928f9bfa0: [0x42001559fae4]vmkWor 1dFunc@vmkernel#nover+0x31 stack: 0x42001559fae0
0x453928f9bfe0: [0x420015adc88e ]CpuSched_StartWor1d@vnkernel#nover+0xbf stack: 0x0
0x453928f9c000: [0x420015544faf ]Debug_IsInitialized@vnkernel#nover0xc stack: 0x0
base fs=0x0 gs=0x420042400000 Kgs=0x0
No disk partition configured to dump data.
Finalized dump header (18/18) FileDump: Successful.
No port for remote debugger. "Escape" for local debugger.

• VMkernel logs (/var/run/log/vmkernel) may have multiple entries listed below, suggesting storage IO failures confirming the scenario. 

nfnic: <1>: INFO: fnic_terminate_ioreq: 2971: io_req: 0x########## sc: 0x########## tag: 0xcd CMD_FLAGS: 0x1000 Found IO in FNIC_IOREQ_CMD_PENDING state on lun

Environment

• VMware vSphere ESXi 8.x
• VMware vSphere ESX 9.x

Cause

This issue is documented/reported by the vendor Cisco and is currently under investigation by their team. 

Cisco Bug: CSCwq55757

Resolution

Contact the hardware vendor for further assistance, potential workarounds, and their official recommendation for a stable alternative to nfnic 5.0.0.46

Additional Information

Uninstalling the driver Instructions for uninstalling third-party VIBs from an ESXi host.

Installing a more stable version via VIB - Patching ESXi host using Command Line