Resolving Host Not Responding State with 'FCPIO_DATA_CNT MISMATCH IO Failure'
search cancel

Resolving Host Not Responding State with 'FCPIO_DATA_CNT MISMATCH IO Failure'

book

Article ID: 389062

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms 

  • The host frequently enters into "not responding" state.
  • Lost access to volume events observed in /var/run/log/hostd.log

2025-07-26T18:48:25.262Z info hostd[2100796] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 8818 : Lost access to volume ########-########-####-############(DS) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

Validation steps: 

  • One or more hosts may be affected by the issue.

  • The driver and firmware versions installed for the impacted adapter are supported with the ESXi version installed in the host and is same as the ones installed on the hosts where there are no issues reported

    For reference on how to check the versions, please refer to Determining Network/Storage firmware and driver version in ESXi.

  • Multiple Aborts observed in Particular path (vmhba2).

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

  • Target-initiated aborts typically occur when Fibre Channel (FC) frames are either dropped or received out of order by the target. The issue indicates problems  beyond ESXi, likely related to the fabric layer connectivity.
  • In this scenario, IOPS are being aborted at the driver layer, which causes the hostd service to crash. When hostd crashes, the host enters a "not responding" state

Cause Validation 

  • Validate the /var/log/vmkernel.log file on the impacted ESXi host and check path failures on specific vmhba (vmhba2) with H:0x7 .This status is returned when a device has been reset due to a Storage Initiator Error       

    2025-02-20T04:51:12.760Z cpu39:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe: 237: NMP device "naa.####################" state in doubt; requested fast path state update. ..
    2025-02-20T04:51:12.760Z cpu39:2098380) ScsiDeviceIO: 4167: Cmd(0x45ba72173948) 0x88, CmdSN 0xffffb684718dcf10 from world 6387083 to dev "naa.##################" failed H:0x7 D:0x0 P:0x0   >>>>>> This status is returned when a device has been reset due to a Storage Initiator Error.
    2025-02-20T04:51:54.754Z cpul:6381388) nfnic: <2>: INFO: fnic_fepio_icmnd_cmpl_handler: 1920: in mismatch actual len: 65536 residual: 0
    2025-02-20T04:51:54.754Z cpul:6381388) WARNING: nfnic: <2>: fnic_fopio_iomnd_cmpl_handler: 1963: so: 0x45####### tag: 0x7f hdr status: FCPIO_DATA_CNT MISMATCH 10 failure!
    2025-02-20T04:51:54.754Z cpu16:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceFrobe:237: NMP device "naa. ##################" state in doubt; requested fast path state update. ..
    2025-02-:26.875Z cpu14:2098380)nfnic: <2>: INFO: fnic_fepio_icmnd_cmpl handler: 1920: in miamatch actual len: 131072 reaidual: 0
    .875Z cpu14:2098380) WARNING: nfnic: <2>: fnic_fepio_icmnd_cmpl_handler: 1963: ac: 0x45######## tag: 0x1b6 hdr atatus: FCPIO DATA CNT MISMATCH IO failure!
    .875Z cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x28 (0x45ba4043c088, O) to dev "naa.####################" on path "vmhba2:CO: T1054:L23" Failed:
    :26.8752 cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 3875: H:0x7 D: 0x0 P:0x0 , Act:EVAL, cmdId.initiator=0x430a426063c0 CmdSN 0x1b
    2:26.8752 cpu14:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa. ####################" state in doubt; requested fast path atate update. ..
    875Z cpu14:2098380) ScsiDeviceIO: 4167: Cmd(0x45ba4043c088) 0x28, CmdSN Oxlb from world 0 to dev "naa.######################" failed H:0x7 D: 0x0 P:0x0
    .876Z cpu14:2098527)nfnic: <2>: INFO: fnic_fopio_icmnd_cmpl_handler: 1920: in mismatch actual len: 131072 residual: 0
    OT04:52:26.876Z cpu14:2098527) WARNING: nfnic: <2>: fnic_fopio_icmnd_cmpl handler: 1963: sc: 0x4######### tag: Oxlbf hdr status: FCPIO_DATA CNT MISMATCH I0 failure!
    .876Z cpul4:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x28 (0x45ba7990e108, O) to dev "naa.#####################" on path "vmhba2:CO: T1059: L47" Failed:
    cpu14:2098380) NMP: nmp_ThrottleLogForDevice: 387S: M:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId. initiator-0x43######## Cmd5N OxldsZ cpu14:2098380) WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa. #####################" state in doubt; requested fast path state update. ..

    2025-02-20T04:53:44.169Z cpul:2098380)nfnic: <2>: INFO: fnic_fopio_iomnd ompl handler: 1920: in mismatch actual len: 65536 residual: 0
    2025-02-20T04:53:44.169Z cpul:2098380) WARNING: nfnic: <2>: fnic_fopio_iomnd_cmpl_handler: 1963: ac: 0x45###### tag: 0x44c hdr atatus: FCPIO_DATA_CNI_MISMATCH IO failure!    >>>>>>>This indicates a mismatch in the data count or alignment during an I/O operation between the host and storage. FCPIO refers to the Fibre Channel Protocol I/O operations, and DATA_CNT MISMATCH suggests there’s an inconsistency between expected and actual data
    2025-02-20T04:53:44.169Z cpul:2098380) NMP: nmp_ThrottleLogForDevice: 3867: Cmd 0x88 (0x45da69935b48, 6387437) to dev "naa. ####################" on path "vmhba2:CO: T1070:L517" Failed:

Resolution

Contact HBA Hardware vendor and validate the reason for IO aborts which is happening in driver layer 

Additional Information

Powered by Wolken Software