iSCSI -- Lost connectivity to Datastore due to iSCSI Nop-out timeouts

Products

• VMware vSphere ESXi

Issue/Introduction

You observe in the ESXi Host Logs the following messages - occurring around the same time:

 

Remark: Per default, ESXi Host Logs reporting in UTC Time.

/var/log/hostd.log

YYYY-MM-DDT19:44:56.049Z In(166) Hostd[######]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 22887 : Lost access to volume ###-###-##-#### (MyDatastore) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

YYYY-MM-DDT19:44:58.707Z In(166) Hostd[######]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 22888 : Successfully restored access to volume ###-###-##-#### (MyDatastore) following connectivity issues.

 

 

/var/log/vmkwarning.log

YYYY-MM-DDT19:44:58.701Z Wa(180) vmkwarning: cpu48:######)WARNING: iscsi_vmk: iscsivmk_StopConnection:736: vmhba##:CH:21 T:0 CN:#: iSCSI connection is being marked "OFFLINE" (Event:4)

 

 

/var/log/syslog.log

YYYY-MM-DDT19:44:58.701Z Wa(28) iscsid[######]: connection 18:0 (iqn.####-##.com.StorageVendor:storage:StorageVendorBox-sn-#### if=iscsi_vmk@vmk## addr=##.##.#.#:3260 (TPGT:#### ISID:0x##)  (T0 C21)) Nop-out timeout after 10 sec in state (3).

YYYY-MM-DDT19:45:33.349Z Wa(28) iscsid[######]: connection 18:0 (iqn.####-##.com.StorageVendor:storage:StorageVendorBox-sn-#### if=iscsi_vmk@vmk## addr=##.##.#.#:3260 (TPGT:#### ISID:0x##)  (T0 C21)) has recovered (# attempts)

 

/var/log/vobd.log

YYYY-MM-DDT19:44:58.707Z In(14) vobd[######]:  [vmfsCorrelator] ###us: [esx.problem.vmfs.heartbeat.recovered] ###-###-##-#### MyDatastore

 

Environment

ESXi (All Versions)

Cause

A NOP-Out timeout occurs if NOP-Outs do not receive a response from the ISCSI target for a period of 10 seconds

NOP-Out is used by the initiator as kind of "ping request" at the iSCSI layer to verify that connection/sessions are still active and all its components are operational and active. 

This operation (NOP OUT) is performed by the initiator to the target.  The NOP-In response is the "ping echo".  A No Operation request sent from the target to the initiator.

 

Resolution

In order to investigate and resolve the Nop-out timeouts please engage with your Network and Storage vendor.