ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"
Article ID: 340039
Updated On:
Products
Issue/Introduction
ESXi host shows "Not Responding" or "Disconnected" in vCenter.
Virtual Machines remain powered on and accessible (Data plane is functional).
Host management agents (`hostd`) are unresponsive to commands.
In `/var/run/log/vmkwarning.log`, the following error is present:
`WARNING: nfnic: <#>: fnic_fcpio_icmnd_cmpl_handler: ####: sc: 0x#### tag: 0x#### hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!`
In '/var/run/log/vmkernel.log', the following errors are present:
2026-07-18T03:47:13.873Z Wa(180) vmkwarning: cpu47:2098148)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1961: sc: 0x45ba016bd640 tag: 0x770 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
2026-07-18T03:47:13.873Z In(182) vmkernel: cpu47:2098148)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x28 (0x45ba016bd640, 3548473) to dev "naa.600a09803831386d7924566#########" on path "vmhba0:C0:T5:L1" Failed:
2026-07-18T03:47:13.873Z In(182) vmkernel: cpu47:2098148)NMP: nmp_ThrottleLogForDevice:3898: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4309fe0a4e00 CmdSN 0x6e
2026-07-18T03:47:13.873Z Wa(180) vmkwarning: cpu47:2098148)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600a09803831386d7924566871726c54" state in doubt; requested fast path state update...
2026-07-18T03:47:13.873Z In(182) vmkernel: cpu47:2098148)ScsiDeviceIO: 4697: Cmd(0x45ba016bd640) 0x28, CmdSN 0x6e from world 3548473 to dev "naa.600a09803831386d7924566#########" failed H:0x7 D:0x0 P:0x0
2026-07-18T03:47:14.039Z In(182) vmkernel: cpu38:2098194)Unmap6: 10551: mcn_na_ucs_arc_dev_W_01: Acquired UC with 0 OP ucOffset 204800 ucIndex: 35
2026-07-18T03:47:14.039Z In(182) vmkernel: cpu38:2098194)Unmap6: 10583: mcn_na_ucs_arc_dev_W_01: Unmap capability 0 OP i: 0 offset: 65536 endROffset: 589824 Acquired: TRUE
2026-07-18T03:47:14.071Z In(182) vmkernel: cpu47:2097817)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1919: in mismatch actual len: 1048576 residual: 0
Storage paths may drop or show as "Dead."
Environment
- VMware vSphere ESXi 7.x
- VMware vSphere ESXi 8.x
Cause
The `FCPIO_DATA_CNT_MISMATCH` error is reported by the Cisco `nfnic` driver when the transmitted data length does not match the expected payload.
This typically indicates physical layer issues (SFP, cabling) or a logic mismatch between the HBA driver and the Fabric Interconnect/UCS firmware.
Resolution
1. Perform a hardware-level hard reboot (cold boot) of the affected ESXi host to clear the management deadlock.
2. Audit the current `nfnic` driver version and Cisco UCS firmware against the VMware Compatibility Guide
3. Ensure the driver and firmware are aligned with the certified combination for your ESXi version.
4. Inspect physical SAN components (SFPs, optical cables) for CRC errors or low light levels.
5. Check the storage array controller logs for hardware faults or failover events occurring at the time of the disconnect.
6. If the issue persists despite certified alignment and healthy physical layer, contact Cisco Support for further hardware diagnostics..
Additional Information
Feedback
