Resolving Host Not Responding State with 'FCPIO_DATA_CNT MISMATCH IO Failure'
search cancel

Resolving Host Not Responding State with 'FCPIO_DATA_CNT MISMATCH IO Failure'

book

Article ID: 389062

calendar_today

Updated On: 04-03-2025

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms 

The host frequently enters a "not responding" state 

Validation steps: 

  • One or more hosts may be affected by the issue.

  • The driver and firmware versions installed for the impacted adapter are supported with the ESXi version installed in the host and is same as the ones installed on the hosts where there are no issues reported

    For reference on how to check the versions, please refer to Determining Network/Storage firmware and driver version in ESXi.

  • Multiple Aborts observed in Particular path (vmhba2)

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

  • Target-initiated aborts typically occur when Fibre Channel (FC) frames are either dropped or received out of order by the target. The issue indicates problems  beyond ESXi, likely related to the fabric layer connectivity.
  • In this scenario, IOPS are being aborted at the driver layer, which causes the hostd service to crash. When hostd crashes, the host enters a "not responding" state

Cause Validation 

  • Validate the /var/log/vmkernel.log file on the impacted ESXi host and check path failures on specific vmhba (vmhba2) with H:0x7 .This status is returned when a device has been reset due to a Storage Initiator Error       

    2025-02-20T04:51:12.760Z cpu39:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: 237: NMP device "naa. 600a################" state in doubt; requested fast path state update. ..
    2025-02-20T04:51:12.760Z cpu39:2098380) ScsiDeviceIO: 4167: Cmd(0x45ba72173948) 0x88, CmdSN 0xffffb684718dcf10 from world 6387083 to dev "naa. 600##############" failed H:0x7 D:0x0 P:0x0   >>>>>> This status is returned when a device has been reset due to a Storage Initiator Error.
    2025-02-20T04:51:54.754Z cpul:6381388) nfnic: <2>: INFO: fnic_fepio_icmnd_cmpl_handler: 1920: in mismatch actual len: 65536 residual: 0
    2025-02-20T04:51:54.754Z cpul:6381388) WARNING: nfnic: <2>: fnic_fopio_iomnd_cmpl_handler: 1963: so: 0x45bas702c788 tag: 0x7f hdr status: FCPIO_DATA_CNT MISMATCH 10 failure!
    2025-02-20T04:51:54.754Z cpu16:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceFrobe:237: NMP device "naa. 600###############" state in doubt; requested fast path state update. ..
    2025-02-:26.875Z cpu14:2098380)nfnic: <2>: INFO: fnic_fepio_icmnd_cmpl handler: 1920: in miamatch actual len: 131072 reaidual: 0
    .875Z cpu14:2098380) WARNING: nfnic: <2>: fnic_fepio_icmnd_cmpl_handler: 1963: ac: 0x45ba4043c088 tag: 0x1b6 hdr atatus: FCPIO DATA CNT MISMATCH IO failure!
    .875Z cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x28 (0x45ba4043c088, O) to dev "naa. 600a09##################763" on path "vmhba2:CO: T1054:L23" Failed:
    :26.8752 cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 3875: H:0x7 D: 0x0 P:0x0 , Act:EVAL, cmdId.initiator=0x430a426063c0 CmdSN 0x1b
    2:26.8752 cpu14:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa. 600##################" state in doubt; requested fast path atate update. ..
    875Z cpu14:2098380) ScsiDeviceIO: 4167: Cmd(0x45ba4043c088) 0x28, CmdSN Oxlb from world 0 to dev "naa. 600####################" failed H:0x7 D: 0x0 P:0x0
    .876Z cpu14:2098527)nfnic: <2>: INFO: fnic_fopio_icmnd_cmpl_handler: 1920: in mismatch actual len: 131072 residual: 0
    OT04:52:26.876Z cpu14:2098527) WARNING: nfnic: <2>: fnic_fopio_icmnd_cmpl handler: 1963: sc: 0x45ba7990e108 tag: Oxlbf hdr status: FCPIO_DATA CNT MISMATCH I0 failure!
    .876Z cpul4:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x28 (0x45ba7990e108, O) to dev "naa. 600###################" on path "vmhba2:CO: T1059: L47" Failed:
    cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 387S: M:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId. initiator-0x430a4laala40 Cmd5N Oxld
    sZ cpu14:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa. 600a09803831374b45244e5642744748" state in doubt; requested fast path state update. ..
    0T04:52:26.876Z cpu14:2098380) ScsiDeviceIO: 4167: Cmd(0x45ba7990e108) 0x28, CmdSN Oxld from world 0 to dev "naa.60##########################" failed H:0x7 D:0x0 P:0x0
    cpu14:6387727) nfnic! <2>: INFO: fnic_fepio_icmnd cmpl handler: 1920: in mismatch actual len: 131072 residual: 0
    Z cpu14: 6387727) WARNING: nfnic: <2>: fnic_fepio_icmnd cmpl handler: 1963: ac: 0x45da69833e48 tag: 0x1d6 hdr status: FCPIO_DATA CNT_MISMATCH IO failure!
    Z cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x28 (0x45da69833e48, O) to dev "naa.600a############################" on path "vmhba2:CO: T1059:L48" Failed:
    3Z cpul4:2098380) NMP: nmp_ThrottleLogForDevice: 3875: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430a4173b300 CmdSN 0x29

    9112 cpu89:6497006) VC: 4721: Refresh open volume time 2 msec
    4.9292 cpu68:6383094)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl handler: 1920: in mismatch actual len: 65536 residual: 0
    2025-02-20T04:53:44.169Z cpul:2098380)nfnic: <2>: INFO: fnic_fopio_iomnd ompl handler: 1920: in mismatch actual len: 65536 residual: 0
    2025-02-20T04:53:44.169Z cpul:2098380) WARNING: nfnic: <2>: fnic_fopio_iomnd_cmpl_handler: 1963: ac: 0x45da69935b48 tag: 0x44c hdr atatus: FCPIO_DATA_CNI_MISMATCH IO failure!    >>>>>>>This indicates a mismatch in the data count or alignment during an I/O operation between the host and storage. FCPIO refers to the Fibre Channel Protocol I/O operations, and DATA_CNT MISMATCH suggests there’s an inconsistency between expected and actual data
    2025-02-20T04:53:44.169Z cpul:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x88 (0x45da69935b48, 6387437) to dev "naa. 600#################" on path "vmhba2:CO: T1070:L517" Failed:

Resolution

Contact HBA Hardware vendor and validate the reason for IO aborts which is happening in driver layer 

Additional Information

Powered by Wolken Software