All shared iSCSI datastores missing from one or more ESXi Hosts
search cancel

All shared iSCSI datastores missing from one or more ESXi Hosts

book

Article ID: 391517

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • All shared iSCSI datastores are missing from a specific ESXi host or set of hosts. 
  • No changes were made to the iSCSI configuration via the vSphere Client
  • Some hosts in the same cluster continue to access the shared iSCSI datastores without issues.
  • From the ESXi host logs, you see the following messages:

    • Repeated warnings where the iSCSI connection is closed by the peer (target), indicating the session is being reset or denied by the storage array:
[from /var/log/vmkernel.log on ESXi host]
 

2025-03-17T07:40:02.756Z cpu26:66509)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: vmhba64:CH:l T:2 CN:0: Failed to receive data: Connection closed by peer
2025-03-17T07:40:02.756Z cpu26:66309)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2025-03-17T07:48:02.756Z cpu26:66389) WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic: Conn [CID: 0 L: <host_vmk_ip>:27237 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:02.756Z cpu26:66389) iscsi_vmk: iscsivmk_ConnRxNotifyFailure: vmhba64:CH:l T:2 CN:0 Connection rx notifying failure: Failed to Receive. State=Bound
2025-03-17T07:40:02.756Z cpu2:66389) iscsi_vmk iscsivmk_ConnRxNotifyFailure: Sess [ISID: TARGET: (null) TPGT: 0 TSIH: 0]
2025-03-17T07:48:02.756Z cpu26:66389) iscsi_vmk: iscsivmk_ConnRxNotifyFailure: Conn [CID: 0 L: <host_vmk_ip>:27237 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:03.009Z cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba64:CH:l T:l CN:0: iSCSI connection is being marked "ONLINE"
2025-03-17T07:48:03.009Z cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000002 TARGET: iqn.1991-05.xxx.xxxxxxx:xxxxxxxxxxx-th-restore-target TPGT: 1 TSIH: 1
2025-03-17T07:48:03.009Z cpu30:69160)WARNING: iscsi_vmk: iscsivmk_StartConnection: Conn [CID: 0 L: <host_vmk_ip>:47456 R: <iscsi_target_ip>:3260]
2025-03-17T07:48:03.010Z cpu30:69160) WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba64:CH:0 T:2 CN:0: iSCSI connection is being marked "OFFLINE" (Event :2)

    • iSCSI login requests are failing with error code 0203, which typically means the iSCSI target is not provisioned or no longer accessible for this initiator:

[from /var/log/vobd.log ]

2025-03-17T07:48:03.010Z: [iscsiCorrelator] 6661830184u8: [vob.iscsi.target.login.error] vmhba64 @ vmk3 failed to login to iqn.1991-05.xxx.xxxxxxx.xxxxxxxxx-ag-nas-target because the target returned a login status of 0203
2025-03-17T07:48:03.010Z: [esx.problem.storage.iscsi.target.login.error] Login to iSCSI target iqn.1991-05.xxx.xxxxxxx.xxxxxxxxx-ag-nas-target on vmhba64 @ vmk3 failed. Target returned login error of:0203.

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x

Cause

  • This issue is caused by improper storage array configuration. Specifically, the iSCSI target has either been deleted or unprovisioned for the affected ESXi host’s initiator. As a result, the host is unable to maintain or re-establish a connection to the shared iSCSI datastores.

Resolution

1. Verify Network Configuration: Ensure the MTU size is consistent across all relevant VMkernel adapters and the associated virtual switch (standard or distributed).

2. Test Connectivity (Jumbo Frames):  Use vmkping to verify that the host can reach the iSCSI target without fragmentation. Run the following command from the ESXi CLI:

vmkping -I <vmk#> -s 1472 -d <iSCSI_target_IP>     (Standard frames)

vmkping -I <vmk#> -s 8972 -d <iSCSI_target_IP>     (Jumbo frames) 

    • Note: The -d flag sets the "Don't Fragment" bit, which is essential for validating MTU path health.

3. Perform a Storage Rescan: From the vSphere Client, trigger a rescan on the affected ESXi host:

    • Right-click host -> Storage -> Rescan Storage.

    • Check the Static Discovery tab of the iSCSI Software Adapter to ensure the target IQN is listed.

4. Validate Storage Array Provisioning: If the datastores do not reappear after a rescan, the issue is likely external to ESXi. Engage your storage vendor to:

    • Verify the iSCSI target is online.

    • Confirm the affected ESXi host’s Initiator IQN is correctly added to the target’s Access Control List (ACL) or Masking View.

5. Re-scan and Verify: Once the storage vendor confirms the mapping is corrected, perform another Rescan Storage on the host to restore the missing datastores.

Additional Information

Powered by Wolken Software