VM's reporting datastore transient connectivity issue with error fcpio hdr status: FCPIO_ITMF_REJECTED' in vmkernel.log
search cancel

VM's reporting datastore transient connectivity issue with error fcpio hdr status: FCPIO_ITMF_REJECTED' in vmkernel.log

book

Article ID: 394461

calendar_today

Updated On: 05-06-2025

Products

VMware vSphere ESXi

Issue/Introduction

Symptom:

  • VMFS volume is intermittently showing lost access to volume events.

 "Lost access to volume <Datastore UUID> (Datastore Name) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly."

Steps to check : Select host > Monitor > Events

Validation:

  • In the '/var/run/log/vobd.log', you will find below entries -

    2025-04-07T04:31:38.394Z In(14) vobd[2097861]:  [scsiCorrelator] 248862363239us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.################################ degraded. Path vmhba2:C0:T553:L31 is down. Affected datastores: Unknown.

2025-04-07T05:53:32.478Z In(14) vobd[2097861]:  [scsiCorrelator] 253753655822us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba2:C0:T560:L31 changed state from on (device ID: naa.################################)

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

  • When a Storage Array rejects an ABORT command, which is Task Management Function (TMF), it typically does this because the command that it attempted to abort never successfully made its way to the storage array, meaning that it was dropped somewhere between the initiator and target. The array cannot abort a command it never received, which is why it rejects the abort request. The Cisco NFNIC driver reacts to this condition by performing an error handling workflow that will quickly logout/login to the array target that rejected the ABORT. This results in a very temporary storage path loss since the logout/login process is very quick. As a result, path loss events will be visible in the ESXi /var/log/vmkernel.log and /var/log/vobd.log, as well as pushed up to vCenter Server.

Since this condition is typically associated with command drops/dropped frames, it is recommended to review the fabric switches and the Cisco Fabric Interconnect (FI) for transmit errors or error counters that could indicate an unhealthy fabric (bad SFP, low light levels, etc). If there are instances of FCPIO_DATA_CNT_MISMATCH in the /var/log/vmkernel.log as well then this is further indication of an unhealthy fabric.

2025-04-07T04:47:40.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1870: io_req: 0x45ba05014c98 sc: 0x45b9e8c16e40 tag: 0x308 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x0$
2025-04-07T04:47:40.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2396: fcpio hdr status: FCPIO_ITMF_REJECTED
2025-04-07T04:47:40.190Z Wa(180) vmkwarning: cpu40:2098052)WARNING: nfnic: <2>: fnic_fcpio_itmf_cmpl_handler: 2427: abort reject received id: 0x308
2025-04-07T04:47:40.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_handle_itmf_reject: 2282: Abort Rejected ! sending TGT_EV_LOGOUT for 0x1400e0
2025-04-07T04:47:40.190Z In(182) vmkernel: cpu40:2098025)nfnic: <2>: INFO: fnic_tport_event_handler: 2105: logging out from tport: 230 tport->fcid: 0x1400e0

  • The Cisco NFNIC driver will print this message when not all of the data expected is transmitted. This typically indicates that out of order frames were received from the array target, which should not happen and are indicative of physical layer issue. Since the FCPIO_DATA_CNT_MISMATCH error is typically associated with a physical layer issue (bad SFP, low light levels, etc), it is recommended to review both the fabric switches and the Cisco Fabric Interconnect (FI) for transmit errors.

    2025-04-07T05:58:18.862Z In(182) vmkernel: cpu74:2098052)NMP: nmp_ThrottleLogForDevice:3898: H:0x1 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x4539b461bb58 CmdSN 0x0
    2025-04-07T05:58:18.862Z Wa(180) vmkwarning: cpu11:2686658)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba2:C0:T560:L31" - No connection (195887168)
    2025-04-07T05:58:18.862Z In(182) vmkernel: cpu11:2686658)nfnic: <2>: INFO: fnic_queuecommand: 734: returning IO as lun is inactive or tport is NULL. driverIO:0
    2025-04-07T05:58:18.862Z Wa(180) vmkwarning: cpu11:2686658)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba2:C0:T552:L31" - No connection (195887168)
    2025-04-07T05:58:49.849Z In(182) vmkernel: cpu59:2098049)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1986: in mismatch actual len: 1048576 residual: 0
    2025-04-07T05:58:49.849Z Wa(180) vmkwarning: cpu59:2098049)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 2028: sc: 0x45d9fc6ada80 tag: 0x705 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!
    2025-04-07T05:58:49.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3842: last error status from device naa.################################ repeated 1 times
    2025-04-07T05:58:49.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x88 (0x45d9fc6ada80, 2101455) to dev "naa.################################" on path "vmhba2:C0:T562:L31" Failed:
    2025-04-07T05:58:49.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3898: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x430eafa3dfc0 CmdSN 0x8000007e
    2025-04-07T05:58:49.849Z Wa(180) vmkwarning: cpu54:2098052)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...
    2025-04-07T05:58:49.849Z In(182) vmkernel: cpu54:2098052)ScsiDeviceIO: 4672: Cmd(0x45d9fc6ada80) 0x88, CmdSN 0x8000007e from world 2101455 to dev "naa.################################" failed H:0x7 D:0x0 P:0x0
    2025-04-07T05:58:49.883Z In(182) vmkernel: cpu77:2097744)nfnic: <2>: INFO: fnic_queuecommand: 734: returning IO as lun is inactive or tport is NULL. driverIO:0
    2025-04-07T05:58:49.883Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x12 (0x45d9fcad6300, 0) to dev "naa.################################" on path "vmhba2:C0:T560:L31" Failed:

Resolution

  • This is a Layer 1 issue and the physical networking/fabric needs to be investigated by network vendor.

Additional Information

Powered by Wolken Software