ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"
Article ID: 340039
Updated On:
Products
Issue/Introduction
ESXi host shows "Not Responding" or "Disconnected" in vCenter.
Virtual Machines remain powered on and accessible (Data plane is functional).
Host management agents (`hostd`) are unresponsive to commands.
In `/var/run/log/vmkwarning.log`, the following error is present:
`WARNING: nfnic: <#>: fnic_fcpio_icmnd_cmpl_handler: ####: sc: 0x#### tag: 0x#### hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!`
In '/var/run/log/vmkernel.log', the following errors are present:
[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu47:2098148)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 1961: sc: 0x45####### tag: 0x#### hdr status: FCPIO_DATA_CNT_MISMATCH IO failure![YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu47:2098148)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x28 (0x45#######, 3548473) to dev "naa.600a######################" on path "vmhba0:C0:T5:L1" Failed:[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu47:2098148)NMP: nmp_ThrottleLogForDevice:3898: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x43####### CmdSN 0x6e[YYYY-MM-DDTHH:MM:SS] Wa(180) vmkwarning: cpu47:2098148)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600####################" state in doubt; requested fast path state update...[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu47:2098148)ScsiDeviceIO: 4697: Cmd(0x45#########) 0x28, CmdSN 0x6e from world 3548473 to dev "naa.600########################" failed H:0x7 D:0x0 P:0x0[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu38:2098194)Unmap6: 10551: mcn_na_ucs_arc_dev_W_01: Acquired UC with 0 OP ucOffset 204800 ucIndex: 35[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu38:2098194)Unmap6: 10583: mcn_na_ucs_arc_dev_W_01: Unmap capability 0 OP i: 0 offset: 65536 endROffset: 589824 Acquired: TRUE[YYYY-MM-DDTHH:MM:SS] In(182) vmkernel: cpu47:2097817)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1919: in mismatch actual len: 1048576 residual: 0
Storage paths may drop or show as "Dead."
Environment
- VMware vSphere ESXi 7.x
- VMware vSphere ESXi 8.x
Cause
The `FCPIO_DATA_CNT_MISMATCH` error is reported by the Cisco `nfnic` driver when the transmitted data length does not match the expected payload.
This typically indicates physical layer issues (SFP, cabling) or a logic mismatch between the HBA driver and the Fabric Interconnect/UCS firmware.
Resolution
1. Perform a hardware-level hard reboot (cold boot) of the affected ESXi host to clear the management deadlock.
2. Audit the current `nfnic` driver version and Cisco UCS firmware against the VMware Compatibility Guide
3. Ensure the driver and firmware are aligned with the certified combination for your ESXi version.
4. Inspect physical SAN components (SFPs, optical cables) for CRC errors or low light levels.
5. Check the storage array controller logs for hardware faults or failover events occurring at the time of the disconnect.
6. If the issue persists despite certified alignment and healthy physical layer, contact Cisco Support for further hardware diagnostics..
Additional Information
Feedback
