Multiple ESXi hosts lose iSCSI storage connectivity simultaneously with VMs unavailable

Products

• VMware vSphere ESXi

Issue/Introduction

• Multiple ESXi hosts lose iSCSI storage connectivity at the same time
  
  
• VMs on those hosts on the iSCSI storage become unavailable
  
  
• Some VMs are found to power off

Environment

VMware vSphere ESXi (all versions)

Cause

• This arises if there is a network or storage array issue causing iSCSI connections to the impacted hosts to go offline.
  
  
• VMs powering off may arise due to HA (High Availability) powering off VMs to restart them on other hosts. However, the VMs fail to power-on because of similar connectivity issues on other hosts.
   
• /var/log/vmkernel.log reports:
  vmkwarning: cpu20:2098182)WARNING: iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:0 T:5 CN:0: iSCSI connection is being marked "OFFLINE" (Event:6)


• For both network and storage issues, /var/log/syslog.log may initially report Nop-out timeout:
  iscsid[2098580] connection 1:0 (iqn.####-##.com.###:########### if=iscsi_vmk@vmk# addr=<targetIP>:3260 (TPGT:32 ISID:0x1)  (T0 C0)) Nop-out timeout after 10 sec in state (3).


• Where there are external network errors /var/log/syslog.log may report:
  iscsid[2098580] Kernel reported iSCSI connection 4:0 error (1011) state (2)
  
  
• Where there are storage issues, /var/log/syslog.log may report:
  iscsid[2098580] connect failed (111,Connection refused)

Resolution

• Investigate and resolve the network/storage issues. 

 

Additional Information

See Troubleshooting ESXi connectivity to iSCSI arrays using software initiators