FNIC aborts on a Cisco server typically indicates an issue with the Fibre Channel (FC) or Fibre Channel over Ethernet (FCoE) connectivity.
Ethernet NIC (eNIC) is the driver for the Cisco UCS VIC Ethernet NIC and Fibre Channel NIC (fNIC) is the driver for the Cisco UCS VIC Fibre Channel over Ethernet HBA or Fiber Channel. enic/fnic drivers are used by ESXi to communicate with the physical HBA/Converged Network Adapter (CNA) installed on the server. These drivers enable FCoE or FC communication between the ESXi host and the target array.
YYYY-MM-DDTHH:MM Wa(10) vmkwarning: cpu##:2097613)WARNING: ScsiPath: 9263: Failed TaskMgmt abort for cmdId.initiator=0x4309c5e9d00 CmdSN 0x1f50, status Status pending, path vmhba#:C#:T#:L#YYYY-MM-DDTHH:MM Wa(10) vmkwarning: cpu##:2097613)WARNING: ScsiPath: 9263: Failed TaskMgmt abort for cmdId.initiator=0x4309c918d40 CmdSN 0x266, status Status pending, path vmhba#:C#:T#:L#YYYY-MM-DDTHH:MM Wa(10) vmkwarning: cpu##:2097613)WARNING: ScsiPath: 9263: Failed TaskMgmt abort for cmdId.initiator=0x4309c5e9d00 CmdSN 0x20e4, status Status pending, path vmhba#:C#:T#:L#YYYY-MM-DDTHH:MM Wa(10) vmkwarning: cpu##:2097613)WARNING: ScsiPath: 9263: Failed TaskMgmt abort for cmdId.initiator=0x4309cb4c9c0 CmdSN 0x51, status Status pending, path vmhba#:C#:T#:L#YYYY-MM-DDTHH:MM cpu##:2097605)WARNING: nfnic: <1>: fnic_abort_cmd: 3826: Abort for cmd tag: 0x2db in pending stateYYYY-MM-DDTHH:MM cpu##:2097605)WARNING: nfnic: <1>: fnic_abort_cmd: 3826: Abort for cmd tag: 0x2f6 in pending stateYYYY-MM-DDTHH:MM cpu##:2117608)WARNING: nfnic: <1>: fnic_abort_cmd: 3826: Abort for cmd tag: 0x224 in pending stateYYYY-MM-DDTHH:MM cpu##:2097604)WARNING: nfnic: <1>: fnic_abort_cmd: 3826: Abort for cmd tag: 0x370 in pending stateYYYY-MM-DDTHH:MM cpu##:2097605)WARNING: nfnic: <1>: fnic_abort_cmd: 3826: Abort for cmd tag: 0x3c4 in pending stateYYYY-MM-DDTHH:MM cpu##:2097604)WARNING: nfnic: <1>: fnic_abort_cmd: 3822: Abort for cmd tag: 0x3c4 already issuedYYYY-MM-DDTHH:MM cpu##:2097604)WARNING: nfnic: <1>: fnic_abort_cmd: 3822: Abort for cmd tag: 0x3c4 already issuedYYYY-MM-DDTHH:MM cpu##:2097604)WARNING: nfnic: <1>: fnic_abort_cmd: 3822: Abort for cmd tag: 0x3c4 already issuedYYYY-MM-DDTHH:MM cpu##:2097604)WARNING: nfnic: <1>: fnic_abort_cmd: 3822: Abort for cmd tag: 0x3c4 already issued
YYYY-MM-DDTHH:MM Wa(180) vmkwarning: cpu##:2097613)WARNING: nfnic: <1>: fnic_abort_cmd: 3818: Abort for cmd tag: 0x36e in pending stateYYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_taskMgmt: 2128: TaskMgmt abort sc->cdb: 0xf1YYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_abort_cmd: 3803: Abort cmd called for Tag: 0x36e issued time: 13003 ms CMD_STATE: FNIC_IOREQ_ABTS_PENDING CDB Opcode: 0xf1 sc:0x45d993bf2cc0 flags: 0x43 lun: 1 target: 0x20b60YYYY-MM-DDTHH:MM Wa(180) vmkwarning: cpu##:2097612)WARNING: nfnic: <1>: fnic_abort_cmd: 3814: Abort for cmd tag: 0x36e already issuedYYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_taskMgmt: 2128: TaskMgmt abort sc->cdb: 0xf1YYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_abort_cmd: 3803: Abort cmd called for Tag: 0x36e issued time: 13003 ms CMD_STATE: FNIC_IOREQ_ABTS_PENDING CDB Opcode: 0xf1 sc:0x45d993bf2cc0 flags: 0x43 lun: 1 target: 0x20b60YYYY-MM-DDTHH:MM Wa(180) vmkwarning: cpu##:2097612)WARNING: nfnic: <1>: fnic_abort_cmd: 3814: Abort for cmd tag: 0x36e already issuedYYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_taskMgmt: 2128: TaskMgmt abort sc->cdb: 0xf1YYYY-MM-DDTHH:MM In(182) vmkernel: cpu##:2097612)nfnic: <1>: INFO: fnic_abort_cmd: 3803: Abort cmd called for Tag: 0x36e issued time: 13003 ms CMD_STATE: FNIC_IOREQ_ABTS_PENDING CDB Opcode: 0xf1 sc:0x45d993bf2cc0 flags: 0x43 lun: 1 target: 0x20b60
VMware vSphere ESXi
These errors can be caused by multiple causes:
A Cisco fnic abort issue needs to be triaged from multiple perspectives (vendors) including the switches, fabric interconnects, storage array and the peripherals like cables and other devices (their configurations) to understand where the problem lies. From ESXi's standpoint we have very limited visibility into what's going on, on the other devices which causes ESXi to generate such responses therefore the root cause of the problem requires thorough analysis and investigation from multiple angles to effectively address the issue:
Review ESXi logs from all hosts in the cluster and find out if we are seeing fnic aborts on a single host or all of them. This is the key in isolating fnic abort scenarios.
Check the HBA driver/firmware versions on all the hosts in the cluster. It's strongly recommended to run on the same version of driver/firmware on all the HBAs of a cluster. Even one odd host in a cluster running a different version of driver/firmware in contrast to others can lead to performance anomalies and instability in the cluster.
Review the Switches, Fabric Interconnect and storage array logs for any significant errors related to the aborts.
Analyze FC/FCoE network topology to see if you are witnessing a change in the usual performance parameters.
Run diagnostic tests on the Cisco UCS Virtual Interface Card (VIC) to check if there's any hardware failure
Run diagnostic tests on the switches to check if there are any port failures and perform a general health check to see whether any of the ports are misbehaving.
Drivers is a software that enables your hardware devices to communicate with your operating system and applications. If your drivers are not compatible with your current system, you might experience errors, crashes, or reduced functionality. Firmware is a software that controls the basic functions of your hardware devices, such as booting, loading & other operations.
If the driver or firmware is incompatible or outdated, you may encounter problems with the hardware or within the operating system (ESXi). And because these devices connect to your network or storage they can have a significant performance impact on your virtual machines overall performance. Therefore, its very critical to check the below information before patching your hosts.
Host hardware compatibility with ESXi version. Contact your server vendor or look up this information on their website.
Driver & firmware compatibility with ESXi version. Look up Broadcom Compatibility Guide to find this information
It's also harmless to upgrade to the highest version of firmware available for your adapters, as long as you are willing to apply this change to a single server and monitor it for a few days before applying it to the cluster.
CAUTION: Flashing the BIOS on a HBA can sometimes lead to unknown hardware problems with the adapter if it's very old or even require a replacement if it kills the hardware. Please proceed to do this update under the supervision of your server manufacturer to be on the safer side.