ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"
Article ID: 340039
Updated On:
Products
Issue/Introduction
ESXi host shows "Not Responding" or "Disconnected" in vCenter.
Virtual Machines remain powered on and accessible (Data plane is functional).
Host management agents (`hostd`) are unresponsive to commands.
In `/var/run/log/vmkwarning.log`, the following error is present:
`WARNING: nfnic: <#>: fnic_fcpio_icmnd_cmpl_handler: ####: sc: 0x#### tag: 0x#### hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!`
Storage paths may drop or show as "Dead."
Environment
- VMware vSphere ESXi 7.x
- VMware vSphere ESXi 8.x
Cause
The `FCPIO_DATA_CNT_MISMATCH` error is reported by the Cisco `nfnic` driver when the transmitted data length does not match the expected payload.
This typically indicates physical layer issues (SFP, cabling) or a logic mismatch between the HBA driver and the Fabric Interconnect/UCS firmware.
Resolution
1. Perform a hardware-level hard reboot (cold boot) of the affected ESXi host to clear the management deadlock.
2. Audit the current `nfnic` driver version and Cisco UCS firmware against the VMware Compatibility Guide
3. Ensure the driver and firmware are aligned with the certified combination for your ESXi version.
4. Inspect physical SAN components (SFPs, optical cables) for CRC errors or low light levels.
5. Check the storage array controller logs for hardware faults or failover events occurring at the time of the disconnect.
6. If the issue persists despite certified alignment and healthy physical layer, contact Cisco Support for further hardware diagnostics..
Additional Information
Feedback
