Unable to detect iSCSI backed VMFS Datastores despite devices being Accessible

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

LUN is visible under Devices, however it is not visible as a datastore.
When trying to "Add Storage", it gives a warning on formatting the LUN before proceeding.
Only the iSCSI LUNs show up on the host connected device list, but they cannot be mounted as VMFS Datastores.
I/O errors may be observed when the iSCSI LUNs are checked with partition information and Hexdump addressing Host busy with Abort commands similar to:

hexdump: /vmfs/devices/disks/naa.6000d310033e28000000000000000003: Device or resource busy

<YYYY-MM-DD>T<time>.628Z cpu22:65920)VMWARE SCSI Id: Could not get disk id for vmhba1:C0:T5:L0</time>

<YYYY-MM-DD>T<time>.196Z cpu36:67172 opID=e85c5cae)WARNING: xpt_scsi_adapter_discover:1224:</time> unable to find target 1: No connection
<YYYY-MM-DD>T<time>.317Z cpu29:68000 opID=55b3417f)vmw_ahci[0000001f]: atapiIssueCommand:Unsupported ScsiCommand: 0x9e, cdblen(0x10) is bigger than limit(0xc)
<YYYY-MM-DD>T<time>.337Z cpu20:65959)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba64:CH:0 T:1 L:1 : Task mgmt "Abort Task" with itt=0x134 (refITT=0x133) timed out.</time>
<YYYY-MM-DD>T<time>.207Z cpu31:66363)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)</time>
<YYYY-MM-DD>T<time>.335Z cpu31:66363)</time>WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000002 TARGET: iqn.2002-03.com.compellent:ID TPGT: 0 TSIH: 0]
<YYYY-MM-DD>T<time>.335Z cpu31:66363)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.123.41:29514 R: 192.168.123.10:3260]
<YYYY-MM-DD>T<time>.419Z cpu29:67375)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d310033e28000000000000000003" state in doubt; requested fast path state update...</time>
<YYYY-MM-DD>T<time>.587Z cpu31:66363)iscsi_vmk:</time> iscsivmk_ConnNetRegister: socket 0x43069f4ed1c0 network resource pool netsched.pools.persist.iscsi associated</time></time>

In the /var/log/vmkernel.log file, you may see similar entries similar to:

YYYY-MM-DDT04:46:45.217Z cpu1:66104)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x439500ced340, 66728) to dev "naa.600140593c9775cd3e97d449edb347dd" on path "vmhba64:C1:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. Act:EVAL
YYYY-MM-DDT04:46:45.217Z cpu1:66104)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140593c9775cd3e97d449edb347dd" state in doubt; requested fast path state update...
YYYY-MM-DDT04:46:45.217Z cpu1:66104)ScsiDeviceIO: 2927: Cmd(0x439500ced340) 0x28, CmdSN 0x5 from world 66728 to dev "naa.600140593c9775cd3e97d449edb347dd" failed H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0.
YYYY-MM-DDT04:46:45.359Z cpu1:65696)NMP: nmp_ThrottleLogForDevice:3546: last error status from device naa.600140593c9775cd3e97d449edb347dd repeated 10 times
YYYY-MM-DDT04:46:45.603Z cpu1:65692)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140593c9775cd3e97d449edb347dd" state in doubt; requested fast path state update...
In the /var/log/vmkwarning.log file, you may see similar entries similar to:

YYYY-MM-DDT04:46:47.602Z cpu1:69214)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140593c9775cd3e97d449edb347dd" state in doubt; requested fast path state update...
YYYY-MM-DDT04:46:48.606Z cpu1:69236)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600140593c9775cd3e97d449edb347dd" state in doubt; requested fast path state update...

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere ESXi 6.x

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This issue occurs when there is a mismatch in the MTU settings defined on either the vSwitch or the vmkernel port group over which the iSCSI interaction is taking place.

To validate if the MTU settings are uniform across the board perform the following steps:

Open an SSH session to the ESXi host as root.
Run this command to list the existing vSwitch configuration.

esxcfg-vswitch -l

For example:

In the above example, there are two port groups defined for iSCSI specific interactions both of which have their MTUs set at 9000.

Note: Its a recommendation widely suggested by most of the Storage Vendors to have the iSCSI specific traffic/interaction taking place over 9000 MTU. However, this is not a mandate and this interaction could also very well run on MTU set for 1500.
Run this command to list out the vmkernel port groups:

esxcfg-vmknic-l

For example:

From the above example, it is evident that there is an MTU mismatch taking place between the vmknic and the vSwitch. The vSwitches have been defined for 9000 MTU whereas the vmkernel portgroups have been adjusted for 1500.

Resolution

To resolve this issue, validate the MTU settings that have been defined across the board, and this would be done using the vmkping command:

vmkping -I <VMkernel Portgroup> <Target IP address > -d -s <MTU size> -c <count>

Note:
For 9000 MTU size, test with packet size of 8972. For 1500 MTU size, test with packet size of 1472.
"-d" flag will forward packets without fragmentation only.

For more information, see Testing VMkernel network connectivity with the vmkping command.

Once the above has been verified, you need to uniformly set the MTU settings to either 9000 or 1500 based on the vmkping command from the output above.

To modify the vSwitch settings through the Web Client, select the host and navigate to Networking > Virtual Switch > iSCSI vSwitch and select Edit.
To modify the VMkernel Portgroup settings through the Web Client, select the host and navigate to Networking > VMkernel Adapters > iSCSI Portgroup and select Edit.

For more information, see:

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ