Storage paths drop with "FCPIO_DATA_CNT_MISMATCH IO failure!" on Cisco VIC adapters during a storage array hardware failure

Products

• VMware vSphere ESXi

Issue/Introduction

 

• ESXi hosts utilizing Cisco UCS Virtual Interface Cards (VIC) with the native nfnic driver experience sudden storage path failures and dead paths.

• vCenter Server triggers multiple Storage Connectivity Alarm alerts for the affected hosts.

• In the /var/run/log/vmkernel.log file, you observe the nfnic driver reporting FCPIO_DATA_CNT_MISMATCH errors, followed by the NMP (Native Multipathing Plugin) marking the storage device state "in doubt":
  
  YYYY-MM-DDTHH:MM:SS.000Z Wa(180) vmkwarning: cpuX:XXXXXXX)WARNING: nfnic: <1>: fnic_fcpio_icmnd_cmpl_handler: 2028: sc: 0x45ba4fac9d40 tag: 0x691 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure! 
YYYY-MM-DDTHH:MM:SS.000Z Wa(180) vmkwarning: cpuX:XXXXXXX)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.600xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" state in doubt; requested fast path state update... 
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpuX:XXXXXXX)ScsiDeviceIO: 4672: Cmd(0x45ba4fac9d40) 0x88, CmdSN 0xe3 from world 7628984 to dev "naa.600xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x7 D:0x0 P:0x0 
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpuX:XXXXXXX)nfnic: <1>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1986: in mismatch actual len: 28672 residual: 0 
...
YYYY-MM-DDTHH:MM:SS.000Z In(182) vmkernel: cpuX:XXXXXXX)NMP: nmp_ResetDeviceLogThrottling:3854: last error status from device naa.600xxxxxxxxxxxxxxxxxxxxxxxxxxxxx repeated 3 times

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x
Cisco UCS Blades

Cause

The FCPIO_DATA_CNT_MISMATCH error is generated by the Cisco nfnic driver when it receives incomplete data or out-of-order frames over the Fibre Channel fabric.

While this error is often attributed to physical layer degradation on the switch fabric (e.g., failing SFPs, bad cables, or switch over-run/under-run conditions), this exact signature also occurs if the target Storage Array experiences a hardware controller failure (such as a faulty DIMM or unexpected controller reboot). When the storage controller abruptly goes offline or panics, inflight I/O frames are truncated or dropped. The ESXi host's Cisco VIC HBA detects the resulting mismatch in the expected frame payload, logs the FCPIO_DATA_CNT_MISMATCH, and drops the path to protect data integrity.

Resolution

To resolve this issue, investigate the physical health of the storage path, starting with the storage target:

1. Check Storage Array Health: Engage your storage administrator or Storage OEM to review the array's hardware logs. Look for unexpected controller failovers, offline controllers, cache vaulting events, or hardware failures (e.g., faulty DIMMs, power supplies, or backplane errors) matching the timestamp of the ESXi path drops.

2. Check Fabric Health: If the storage array is completely healthy, inspect the intermediate physical fabric layer. Review the Cisco Fabric Interconnects (FI) and intermediate FC switches for transmit/receive errors, CRC errors, or low optical light levels on the SFPs servicing the ESXi host and the storage array.

Additional Information

For general troubleshooting of Cisco VIC physical layer storage issues, see:

• ESXi host accessing storage via Cisco VIC HBA might go into not responding due to loss of storage access (2149522)