Error: Lost access to volume <Datastore UUID> (Datastore Name) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly. Resulting in transient datastore connectivity issues
Article ID: 394461
Updated On:
Products
Issue/Introduction
- VMFS volume intermittently logs 'lost access' events.
Lost access to volume <Datastore UUID> (Datastore Name) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly. - Events are verified by navigating to the ESXi host > Monitor > Events tab.
- The following entries are recorded in the
/var/run/log/vobd.log:YYYY-MM-DDTHH:MM:SS.394Z In(14) vobd[2097861]: [scsiCorrelator] 248862363239us: [esx.problem.storage.redundancy.degraded] Path redundancy to storage device naa.################################ degraded. Path vmhba2:C0:T553:L31 is down. Affected datastores: Unknown.YYYY-MM-DDTHH:MM:SS.478Z In(14) vobd[2097861]: [scsiCorrelator] 253753655822us: [vob.scsi.scsipath.pathstate.deadver2] scsiPath vmhba2:C0:T560:L31 changed state from on (device ID: naa.################################) - The following entries are recorded in the
/var/run/log/vmkernel.log:YYYY-MM-DDTHH:MM:SS.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1870: io_req: 0x45ba05014c98 sc: 0x45b9e8c16e40 tag: 0x308 CMD_FLAGS: 0x53 CMD_STATE: FNIC_IOREQ_ABTS_PENDING ABTS pending hdr status: FCPIO_ABORTED scsi_status: 0x0$YYYY-MM-DDTHH:MM:SS.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_fcpio_itmf_cmpl_handler: 2396: fcpio hdr status: FCPIO_ITMF_REJECTEDYYYY-MM-DDTHH:MM:SS.190Z Wa(180) vmkwarning: cpu40:2098052)WARNING: nfnic: <2>: fnic_fcpio_itmf_cmpl_handler: 2427: abort reject received id: 0x308YYYY-MM-DDTHH:MM:SS.190Z In(182) vmkernel: cpu40:2098052)nfnic: <2>: INFO: fnic_handle_itmf_reject: 2282: Abort Rejected ! sending TGT_EV_LOGOUT for 0x1400e0YYYY-MM-DDTHH:MM:SS:40.190Z In(182) vmkernel: cpu40:2098025)nfnic: <2>: INFO: fnic_tport_event_handler: 2105: logging out from tport: 230 tport->fcid: 0x1400e0YYYY-MM-DDTHH:MM:SS.862Z In(182) vmkernel: cpu74:2098052)NMP: nmp_ThrottleLogForDevice:3898: H:0x1 D:0x0 P:0x0 . Act:NONE. cmdId.initiator=0x4539b461bb58 CmdSN 0x0YYYY-MM-DDTHH:MM:SS.862Z Wa(180) vmkwarning: cpu11:2686658)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba2:C0:T560:L31" - No connection (195887168)YYYY-MM-DDTHH:MM:SS.862Z In(182) vmkernel: cpu11:2686658)nfnic: <2>: INFO: fnic_queuecommand: 734: returning IO as lun is inactive or tport is NULL. driverIO:0YYYY-MM-DDTHH:MM:SS:18.862Z Wa(180) vmkwarning: cpu11:2686658)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:190: Could not get page 83 INQUIRY data for path "vmhba2:C0:T552:L31" - No connection (195887168)YYYY-MM-DDTHH:MM:SS:49.849Z In(182) vmkernel: cpu59:2098049)nfnic: <2>: INFO: fnic_fcpio_icmnd_cmpl_handler: 1986: in mismatch actual len: 1048576 residual: 0YYYY-MM-DDTHH:MM:SS:49.849Z Wa(180) vmkwarning: cpu59:2098049)WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 2028: sc: 0x45d9fc6ada80 tag: 0x705 hdr status: FCPIO_DATA_CNT_MISMATCH IO failure!YYYY-MM-DDTHH:MM:SS:49.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3842: last error status from device naa.################################ repeated 1 timesYYYY-MM-DDTHH:MM:SS:49.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x88 (0x45d9fc6ada80, 2101455) to dev "naa.################################" on path "vmhba2:C0:T562:L31" Failed:YYYY-MM-DDTHH:MM:SS.849Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3898: H:0x7 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x########## CmdSN 0x8000007eYYYY-MM-DDTHH:MM:SS.849Z Wa(180) vmkwarning: cpu54:2098052)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.################################" state in doubt; requested fast path state update...YYYY-MM-DDTHH:MM:SS:49.849Z In(182) vmkernel: cpu54:2098052)ScsiDeviceIO: 4672: Cmd(0x45d9fc6ada80) 0x88, CmdSN 0x8000007e from world 2101455 to dev "naa.################################" failed H:0x7 D:0x0 P:0x0YYYY-MM-DDTHH:MM:SS.883Z In(182) vmkernel: cpu77:2097744)nfnic: <2>: INFO: fnic_queuecommand: 734: returning IO as lun is inactive or tport is NULL. driverIO:0YYYY-MM-DDTHH:MM:SS:49.883Z In(182) vmkernel: cpu54:2098052)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x12 (0x45d9fcad6300, 0) to dev "naa.################################" on path "vmhba2:C0:T560:L31" Failed:
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
The issue is caused by a command being dropped between the initiator and the storage array target, resulting in the original command not being received by the storage array. As a result, the subsequent ABORT Task Management Function (TMF) request cannot be processed and is rejected by the target.
Upon detection of the ABORT rejection, a target logout/login recovery workflow is initiated by the Cisco NFNIC driver. During this recovery process, a brief storage path interruption may occur, and path loss events may be logged in the ESXi
vmkernelandvobdlogs and reported to the vCenter Server.This condition is commonly associated with dropped commands, dropped Fibre Channel frames, or other SAN fabric communication issues. The FCPIO_DATA_CNT_MISMATCH message is generated when the amount of data received does not match the expected data count because Fibre Channel frames are delivered out of order. Since out-of-order frame delivery is not expected during normal Fibre Channel operation, this condition generally indicates an underlying SAN fabric or physical-layer issue, such as faulty SFPs, degraded optical signal levels, cabling faults, or other transmission errors.
Resolution
This indicates a Layer 1 (physical) issue. Engage the network vendor to investigate the physical network fabric.
Additional Information
Feedback
