Dead path/Path loss may be reported on vCenter Server and ESXi when the Cisco NFNIC Driver reports FCPIO_ITMF_REJECTED

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware vSphere ESXi 5.x - View
• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

The following is reported in /var/log/vmkernel.log on the ESXi Server:

2025-01-15T07:03:17.069Z cpu122:2098489)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1809: io_req: 0x45ea4720d6a0 sc: 0x45ea6766a088 tag: 0x6fd CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x$
2025-01-15T07:03:17.069Z cpu122:2098489)nfnic: <1>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED <--- Storage array target has rejected the ABORT command
2025-01-15T07:03:17.069Z cpu122:2098489)WARNING: nfnic: <1>: fnic_fcpio_itmf_cmpl_handler: 2360: abort reject received id: 0x6fd
2025-01-15T07:03:17.069Z cpu122:2098489)nfnic: <1>: INFO: fnic_handle_itmf_reject: 2215: Abort Rejected ! sending TGT_EV_LOGOUT for 0x7e0040 <--- Cisco NFNIC driver performing target logout/login sequence

Cause

When a storage array rejects an ABORT command, which is Task Management Function (TMF), it typically does this because the command that it attempted to abort never successfully made its way to the storage array, meaning that it was dropped somewhere between the initiator and target. The array cannot abort a command it never received, which is why it rejects the abort request. The Cisco NFNIC driver reacts to this condition by performing an error handling workflow that will quickly logout/login to the array target that rejected the ABORT. This results in a very temporary storage path loss since the logout/login process is very quick. As a result, path loss events will be visible in the ESXi /var/log/vmkernel.log and /var/log/vobd.log, as well as pushed up to vCenter Server.

Resolution

Since this condition is typically associated with command drops/dropped frames, it is recommended to review the fabric switches and the Cisco Fabric Interconnect (FI) for transmit errors or error counters that could indicate an unhealthy fabric (bad SFP, low light levels, etc). If there are instances of FCPIO_DATA_CNT_MISMATCH in the ESXi /var/log/vmkernel.log as well then this is further indication of an unhealthy fabric.