ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• ESXi host with Cisco VIC HBA is in unresponsive state.
• hostd process is unresponsive.
• LUNs are disconnected and fails to boot using FC/FCoE.
• VM replication getting failed while using external storage replication tool.
• In the file /var/run/log/vmkwarning.log you see errors similar to,

YYYY-MM-DDThh:mm:ss.msZ cpu28:33310)WARNING: LinScsi: SCSILinuxAbortCommands:1890: Failed, Driver fnic, for vmhba4
YYYY-MM-DDThh:mm:ss.msZ  cpu27:47509)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 512, Resid = 131072
YYYY-MM-DDThh:mm:ss.msZ  cpu27:47509)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.##############" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ  cpu17:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.##############" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ cpu3:40875)WARNING: VMotion: 1884: 1489551966631519 S: Waited 119.995 seconds for the monitor to process a preCopyNext action. This may cause unexpected vMotion failures.
YYYY-MM-DDThh:mm:ss.msZ  cpu25:33529)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 4096, Resid = 1048576
YYYY-MM-DDThh:mm:ss.msZ  cpu25:33529)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.##############" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ  cpu46:33310)WARNING: LinScsi: SCSILinuxAbortCommands:1890: Failed, Driver fnic, for vmhba4

• In the /var/run/log/vmkernel.log file, you see errors similar to:

YYYY-MM-DDThh:mm:ss.msZ cpu32:33284)<3>fnic : 4 :: hdr status = FCPIO_DATA_CNT_MISMATCH
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 512, Resid = 131072
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x2a (0x439e58573580, 32830) to dev "naa.##############" on path "vmhba4:C0:T4:L49" Failed: H:0x7 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.##############" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)ScsiDeviceIO: 2613: Cmd(0x439e58573580) 0x2a, CmdSN 0x677 from world 32830 to dev "naa.##############" failed H:0x7 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0.
YYYY-MM-DDThh:mm:ss.msZ cpu32:33284)<3>fnic : 4 :: hdr status = FCPIO_DATA_CNT_MISMATCH

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

• This issue occurs due to a bug with Cisco VIC hardware adapter.
• Physical Layer issues on the path to storage
• For more information, see Cisco Bug: CSCva47085 - VIC1340+2304 IOM Native 40g Link Training Issue Causes Connectivity Loss.

Resolution

• This is not a Broadcom issue.
• To resolve this issue, contact Cisco support.
  • Note: This defect was fixed in 3.1(2b), which released in the Fall of 2016.