Cisco NFNIC Driver reporting FCPIO_DATA_CNT_MISMATCH in /var/log/vmkernel.log

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 5.0
• VMware vSphere ESXi 5.5
• VMware vSphere ESXi 5.x - View
• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

When reviewing the vmkernel.log on an ESXi host, the following event is observed:

2025-02-06T16:04:07.633Z Wa(180) vmkwarning: cpu63:2103150)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da6044e2c0 tag: 0x69a hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:04:14.213Z Wa(180) vmkwarning: cpu53:2097877)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da612f9440 tag: 0x646 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:06:33.347Z Wa(180) vmkwarning: cpu51:2106251)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da528c1e80 tag: 0x1c9 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:07:38.938Z Wa(180) vmkwarning: cpu32:2101589)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da613fd240 tag: 0x649 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:07:38.981Z Wa(180) vmkwarning: cpu32:2103953)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da5285cd80 tag: 0x662 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:07:44.346Z Wa(180) vmkwarning: cpu43:2097878)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da58dde300 tag: 0x25b hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:07:45.308Z Wa(180) vmkwarning: cpu43:2104431)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da58d0f400 tag: 0x463 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:08:09.770Z Wa(180) vmkwarning: cpu59:2101584)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da528c5580 tag: 0x72d hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2025-02-06T16:09:10.819Z Wa(180) vmkwarning: cpu36:2097878)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1964: sc: 0x45da58c11e80 tag: 0x70e hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!

Cause

The Cisco NFNIC driver will print this message when not all of the data expected is transmitted. This typically indicates that out of order frames were received from the array target, which should not happen and are indicative of physical layer issue.

Resolution

Since the FCPIO_DATA_CNT_MISMATCH error is typically associated with a physical layer issue (bad SFP, low light levels, etc), it is recommended to review both the fabric switches and the Cisco Fabric Interconnect (FI) for transmit errors.