ESXi host loses storage access intermittently with 'Lost access to volume

Products

VMware vSphere ESXi

Issue/Introduction

ESXi hosts connected to NetApp iSCSI storage arrays may experience intermittent but frequent volume disconnects. The vCenter Server triggers alerts indicating lost access to specific volumes, while the underlying ESXi logs show target-initiated connection drops, TCP resets, and standard connection failure codes. This article provides the cause for these abrupt storage network disconnections and the necessary steps to restore reliable iSCSI communication between the host and the storage target.

Symptoms

The vCenter Server UI displays the following alert: Lost access to volume #### due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
The /var/log/hostd.log file contains log entries similar to these:
info 'Vimsvc.ha-eventmgr'] Event 205 : Lost access to volume #######-###-###-###### due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
info 'Vimsvc.ha-eventmgr'] Event 210 : Successfully restored access to volume #######-###-###-###### following connectivity issues.
Additionally, the /var/log/vobd.log file may contain log entries such as:
CPU##:######)WARNING: LinScsi: SCSILinuxAbortCommands:1891: Failed, Driver fnic, for vmhba1
CPU##:######)<7>fnic : 2 :: Abort Cmd called FCID 0x640380, LUN 0xXX TAG bd flags 3
CPU##:######)<6>fnic : 2 :: abort reject recd. id 189
CPU##:######)<7>fnic : 2 :: abts cmpl recd. id 189 status FCPIO_ITMF_REJECTED
CPU##:######)<7>fnic : 2 :: Returning from abort cmd type 2 FAILED
CPU##:######)WARNING: LinScsi: SCSILinuxAbortCommands:1891: Failed, Driver fnic, for vmhbaX
CPU##:######)<7>fnic : 2 :: Abort Cmd called FCID 0x640380, LUN 0x65 TAG bf flags 3
CPU##:######)<6>fnic : 2 :: abort reject recd. id 191
CPU##:######)<7>fnic : 2 :: abts cmpl recd. id 191 status FCPIO_ITMF_REJECTED
CPU##:######)<7>fnic : 2 :: Returning from abort cmd type 2 FAILED
The /var/run/log/vobd.log shows the storage target terminating the session and the connection stopping:
####-##-##T##:##:##.###Z In(14) vobd[####]: [iscsiCorrelator] ####us: [vob.iscsi.target.async.event] The target iqn.####.com.netapp:#### issued an async event for vmhba64 with reason code 0002
####-##-##T##:##:##.###Z In(14) vobd[####]: [iscsiCorrelator] ####us: [vob.iscsi.connection.stopped] iScsi connection 0 stopped for vmhba64:C1:T0
####-##-##T##:##:##.###Z In(14) vobd[####]: [vmfsCorrelator] ####us: [vob.vmfs.heartbeat.timedout] #### ESX02
The /var/run/log/syslog.log shows the ESXi host experiencing target-side disconnects:
####-##-##T##:##:##.###Z Wa(28) iscsid[####]: Target dropping connection 0 on iqn.####.com.netapp:#### if=default addr=####:3260 (TPGT:2 ISID:0x2) (T0 C1), reconnect min 5 max 0
####-##-##T##:##:##.###Z Wa(28) iscsid[####]: Kernel reported iSCSI connection 4:0 error (1011) state (3)
The /var/run/log/vmkernel.log shows iSCSI connections flapping and network send/receive failures:
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu29:####)WARNING: iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu29:####)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:478: vmhba64:CH:1 T:0 CN:0: Failed to receive data: Connection reset by peer
####-##-##T##:##:##.###Z In(182) vmkernel: cpu0:####)iscsi_vmk: iscsivmk_ConnTxNotifyFailure:1185: vmhba64:CH:1 T:0 CN:0: Connection tx notifying failure: Network Send Error. State=Recovery
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu35:####)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue:637: vmhba64:CH:1 T:0 L:2 : Task mgmt "Abort Task" with itt=0x259b65 (refITT=0x259b5c) timed out.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

The "Lost access to volume" alert is a symptom of an network disconnection between the ESXi host and the storage array. The storage target sends a TCP RST (Reset) packet, abruptly severing the iSCSI connection. Because the disconnection is ungraceful, the software iSCSI adapter transitions to a recovery state, leading to a cascade of I/O failures and path timeouts. This behavior is often associated with outdated network interface card (NIC) drivers or firmware, or non-uniform MTU configurations within the network environment.

Resolution

To stabilize the storage network and prevent target-initiated session terminations, execute the following steps:

Engage Hardware Vendor validate and upgrade the ESXi host's network interface card (NIC) firmware and drivers to the latest supported versions .Aligning the host's network stack with the Hardware Compatibility List (HCL) ensures the ESXi kernel handles TCP segmentation and iSCSI signaling correctly.
Engage NetApp Support to analyze the storage array logs. Provide the vendor with the exact timestamps of the disconnects found in the vobd.log and syslog.log to determine why the array is issuing Async Event 0002 and TCP RST packets to the ESXi initiators.
Review the physical network path—including all switches, routers, and firewalls—located between the ESXi hosts and the NetApp array. Ensure no intermediate network devices are configured to abruptly tear down idle or active TCP sessions.

Additional Information

Broadcom VMware Compatibility Guide

For assistance from Broadcom Support, navigate to: Contact Broadcom Support