Issues with Network Connectivity and FC Access after Cisco Fabric Interconnect upgrade

Products

• VMware vSphere ESXi

Issue/Introduction

Network connectivity issues and iSCSI disconnects after Cisco Fabric Interconnect upgrade

The following messages can be seen in the vmkernel.log on the hosts impacted by this issue. 

FCPIO_ITMF_REJECTED Messages 

2025-03-17T04:32:25.011Z cpu88:2513243)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-03-17T04:32:25.011Z cpu88:2513243)nfnic: <2>: INFO: fnic_handle_itmf_reject: 2215: Abort Rejected ! sending TGT_EV_LOGOUT for 0x9f0060
2025-03-17T09:02:40.725Z cpu64:2101023)nfnic: <1>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-03-17T09:02:40.725Z cpu64:2101023)nfnic: <1>: INFO: fnic_handle_itmf_reject: 2215: Abort Rejected ! sending TGT_EV_LOGOUT for 0x5c0000
2025-03-17T09:03:42.578Z cpu107:2098330)nfnic: <1>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-03-17T09:03:42.578Z cpu107:2098330)nfnic: <1>: INFO: fnic_handle_itmf_reject: 2215: Abort Rejected ! sending TGT_EV_LOGOUT for 0x5c0000
2025-03-17T09:04:45.570Z cpu82:2097745)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-03-17T09:04:45.570Z cpu82:2097745)nfnic: <2>: INFO: fnic_handle_itmf_reject: 2215: Abort Rejected ! sending TGT_EV_LOGOUT for 0x9f0060
2025-03-17T09:04:45.570Z cpu82:2097745)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-03-17T09:04:45.570Z cpu82:2097745)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2328: fcpio hdr status: FCPIO_ITMF_REJECTED

FCPIO_DATA_CNT_MISMATCH IO failure Messages 

2025-03-18T02:23:55.699Z cpu56:3035836)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1920: in mismatch actual len: 1048576 residual: 0
2025-03-18T02:23:55.699Z cpu56:3035836)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1963: sc: 0x45b95d60ed08 tag: 0x217 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-03-18T02:23:55.699Z cpu80:2098330)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x28 (0x45b95d60ed08, 2099705) to dev "naa.###################" on path "vmhba0:C0:T121:L6" Failed:
2025-03-18T02:23:55.699Z cpu80:2098330)NMP: nmp_ThrottleLogForDevice:3875: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430881408fc0 CmdSN 0x170
2025-03-18T02:23:55.699Z cpu80:2098330)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.###################" state in doubt; requested fast path state update...
2025-03-18T02:23:55.699Z cpu80:2098330)ScsiDeviceIO: 4176: Cmd(0x45b95d60ed08) 0x28, CmdSN 0x170 from world 2099705 to dev "naa.###################" failed H:0x7 D:0x0 P:0x0

Cause

• The Cisco NFNIC driver will print this message when not all of the data expected is transmitted. This typically indicates that out of order frames were received from the array target, which should not happen and are indicative of physical layer issue.
• This is a Layer 1 issue and the physical networking/fabric needs to be investigated by network/storage team/vendor.

Resolution

See below KBs for further details. 

Dead path/Path loss may be reported in vCenter Server as well as /var/log/vmkernel.log on ESXi when the Cisco NFNIC Driver reports FCPIO_ITMF_REJECTED

ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"