VMs are down on host that is unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

During normal operation all VMs are found hung on a single host in a cluster and the host must be force rebooted. 

Environment

ESXi (all versions)

Cisco UCS with nfnic driver prior to 5.0.0.48

Cause

A storage driver deadlock occurred.  

While commands were in flight the driver reset the links resulting in a new fabric login. The driver then attempts to abort the commands as they have timed out, but receives no response and attempts again. This double abort results in a deadlock condition due to a driver issue.

You may similar to the following in the vmkernel log, note that the logs do not indicate path loss before the reset and new login.

2026-01-28T20:40:07.831Z In(182) vmkernel: cpu0:2098004)nfnic: <1>: INFO: fdls_tgt_send_adisc: 1316: sending ADISC to tgt: 0x10700
2026-01-28T20:40:07.831Z In(182) vmkernel: cpu0:2098004)nfnic: <1>: INFO: fdls_tgt_send_adisc: 1316: sending ADISC to tgt: 0x10600
2026-01-28T20:40:07.831Z In(182) vmkernel: cpu0:2098004)nfnic: <1>: INFO: fdls_process_gpn_ft_rsp: 2626: iport->state: 4
2026-01-28T20:40:07.831Z In(182) vmkernel: cpu0:2098004)nfnic: <1>: INFO: fdls_process_tgt_adisc_rsp: 2310: ADISC accepted from target: 0x11601. TGT now in ready state. Target logged in
2026-01-28T20:40:07.831Z In(182) vmkernel: cpu0:2098004)nfnic: <1>: INFO: fdls_process_tgt_adisc_rsp: 2310: ADISC accepted from target: 0x11801. TGT now in ready state. Target logged in

2026-01-28T20:41:03.377Z In(182) vmkernel: cpu17:2098077)nfnic: <2>: INFO: fnic_abort_cmd: 3862: Abort cmd called for Tag: 0x263 issued time: 15309 ms CMD_STATE: FNIC_IOREQ_ABTS_PENDING CDB Opcode: 0x2a sc:0x45da4735d2c0 flags: 0x43 lun: 246 target: 0x10200
2026-01-28T20:41:03.377Z Wa(180) vmkwarning: cpu17:2098077)WARNING: nfnic: <2>: fnic_abort_cmd: 3873: Abort for cmd tag: 0x263 already issued

Please see the Additional Information section for details on the nfnic driver.

Resolution

Contact Cisco for the recommended driver and firmware to use to avoid this issue. 

Additional Information

Please see the following:

• Cisco CSCwo37392
• Cisco CSCwq55757
• "Path redundancy to storage device degraded" alerts intermittently received in Cisco UCS environment.
• PSOD crash on CISCO UCS Servers - Backtrace fnic_queue_abort_io_req & fnic_terminate_ioreq
• Temporary/transient storage path loss on Host could result in paths not coming back when using Cisco UCS and NFNIC
• Cisco FAQ: FNIC Aborts
• Broadcom KB: Cisco UCS servers log FNIC aborts in ESXi