ESXi host with Cisco HBA driver fails with the error: "FCPIO_DATA_CNT_MISMATCH"

Products

• VMware vSphere ESXi

Issue/Introduction

 

• ESXi host with Cisco VIC HBA is in unresponsive state.
• hostd process is unresponsive.
• LUNs are disconnected and fails to boot using FC/FCoE.
• In the file /var/run/log/vmkwarning.log you see errors similar to:

         

YYYY-MM-DDThh:mm:ss.msZ cpu28:33310)WARNING: LinScsi: SCSILinuxAbortCommands:1890: Failed, Driver fnic, for vmhba4
YYYY-MM-DDThh:mm:ss.msZ  cpu27:47509)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 512, Resid = 131072
YYYY-MM-DDThh:mm:ss.msZ  cpu27:47509)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.514f0c59bf000079" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ  cpu17:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.514f0c59bf0000a9" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ cpu3:40875)WARNING: VMotion: 1884: 1489551966631519 S: Waited 119.995 seconds for the monitor to process a preCopyNext action. This may cause unexpected vMotion failures.
YYYY-MM-DDThh:mm:ss.msZ  cpu25:33529)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 4096, Resid = 1048576
YYYY-MM-DDThh:mm:ss.msZ  cpu25:33529)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.514f0c59bf0000a9" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ  cpu46:33310)WARNING: LinScsi: SCSILinuxAbortCommands:1890: Failed, Driver fnic, for vmhba4

 

• In the /var/run/log/vmkernel.log file, you see errors similar to:

 

YYYY-MM-DDThh:mm:ss.msZ cpu32:33284)<3>fnic : 4 :: hdr status = FCPIO_DATA_CNT_MISMATCH
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)WARNING: LinScsi: SCSILinuxProcessCompletions:826: Error BytesXferred > Requested Length Marking transfer length as 0 - vmhba = vmhba4, Driver Name = fnic, Requested length = 512, Resid = 131072
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x2a (0x439e58573580, 32830) to dev "naa.514f0c59bf0000a7" on path "vmhba4:C0:T4:L49" Failed: H:0x7 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.514f0c59bf0000a7" state in doubt; requested fast path state update...
YYYY-MM-DDThh:mm:ss.msZ cpu24:33308)ScsiDeviceIO: 2613: Cmd(0x439e58573580) 0x2a, CmdSN 0x677 from world 32830 to dev "naa.514f0c59bf0000a7" failed H:0x7 D:0x28 P:0x0 Possible sense data: 0x0 0x0 0x0.
YYYY-MM-DDThh:mm:ss.msZ cpu32:33284)<3>fnic : 4 :: hdr status = FCPIO_DATA_CNT_MISMATCH

Cause

This issue occurs due to a bug with Cisco VIC hardware adapter.

For more information, see Cisco Bug: CSCva47085 - VIC1340+2304 IOM Native 40g Link Training Issue Causes Connectivity Loss.

Resolution

This is not a VMware issue.

To resolve this issue, contact Cisco support.