All shared iSCSI datastores missing from a single ESXi Host

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• All shared iSCSI datastores are missing from a specific ESXi host.

• No changes were made to the iSCSI configuration via the vSphere Client

• Other hosts in the same cluster continue to access the shared iSCSI datastores without issues.

Validation Steps

• Ensure the MTU size is consistent across all relevant VMkernel adapters and the associated virtual switch.

• Use vmkping to test connectivity from the affected host to the iSCSI target with the specified MTU. The command format is vmkping -I <vmk#> -s <MTU> -d <iSCSI_target_IP>

• From the vSphere Client, perform a storage adapter rescan on the affected ESXi host. Validate whether the shared iSCSI datastores are rediscovered after the rescan.

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

Cause

This issue is caused by improper storage array configuration. Specifically, the iSCSI target has either been deleted or unprovisioned for the affected ESXi host’s initiator. As a result, the host is unable to maintain or re-establish a connection to the shared iSCSI datastores.

Cause Validation

The /var/log/vmkernel.log file shows repeated warnings where the iSCSI connection is closed by the peer (target), indicating the session is being reset or denied by the storage array:

2025-03-17T07:40:02.756Z cpu26:66509)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba64:CH:l T:2 CN:O: Failed to receive data: Connection closed by peer
2025-03-17T07:40:02.756Z cpu26:66309)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: O TSIH: 0]
2025-03-17T07:48:02.7562 cpu26:66389) WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: <host_vmk_ip>:27237 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:02.756Z cpu26:66389) iscsi_vmk: iscsivnk_ConnRxNotifyFailure: vmhba64:CH:l T:2 CN:01 Comnection rx notifying failure: Failed to Receive. State=Sound
2025-03-17T07:40:02.756 cpu2:66389) iscsi_vmk iscsivmk_ConnRxNotifyFailure: Seas [ISID: TARGET: (null) TPGT: O TSIH: O]
2025-03-17T07:48:02.756Z cpu26:66389) iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: <host_vmk_ip>:27237 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:03.0092 cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba64:CH:l T:l CN:O: iSCSI connection is being marked "ONLINE"
2025-03-17T07:48:03.0092 cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn. 1991-05.xxx.xxxxxxx:xxxxxxxxxxx-th-restore-targes TPGT: 1 TSIH: 01
2025-03-17T07:48:03.009Z cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: O L: <host_vmk_ip>:47456 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:03.010Z cpu30:69160) WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:0 T:2 CN:0: iSCSI connection is being marked "OFFLINE" (Event :2)

The /var/log/vobd.log file entries indicate that the iSCSI login request is failing with error code 0203, which typically means the iSCSI target is not provisioned or no longer accessible for this initiator:

2025-03-17T07:48:03.010Z: [iscsiCorrelator] 6661830184u8: [vob.iscsi.target.login.error] vmhba64 @ vmk3 failed to login to iqn.1991-05.xxx.xxxxxxx.xxxxxxxxx-ag-nas-target because the target returned a login status of 0203
2025-03-1707:48:03.010Z: [esx.problem.storage.iscsi.target.login.error] Login to iSCSI target iqn.1991-05.xxx.xxxxxxx.xxxxxxxxx-ag-nas-target on vmhba64 @ vmk3 failed. Target returned login error of:0203.

 

Resolution

Engage your storage vendor to review the configuration of the iSCSI targets on the storage array.

Ensure that the affected ESXi host’s iSCSI initiator (IQN) is properly provisioned and has access to the necessary targets.

Once the configuration is corrected, perform a rescan on the host’s storage adapters to restore connectivity to the shared datastores.

Additional Information

Target information not updated by iSCSI array