VM guest iSCSI initiator intermittently loses connection to vSAN iSCSI target

Products

• VMware vSAN

Issue/Introduction

• vSAN iSCSI targets are configured on a cluster and a number of VMs are configured to access specific targets
  
  
• On one or more VMs, the VM guest iSCSI initiator intermittently loses connection to the vSAN iSCSI target, for which it has access configured, because the TCP connection is closed
  
  
• On losing connection, the initiator reconnects to the target within a few seconds via another host (via vmkernel port configured for vSAN iSCSI target)
  
  
• The VM guest reports logging similar to:

iscsid: Kernel reported iSCSI connection 7:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3)
iscsid: session 7 re-opening (reopen_cnt 0)
iscsid: session 7 disconnecting conn 0x55e0293a4348, fd 22
iscsid: session 7 re-opening (reopen_cnt 0)
iscsid: session 7 connecting to <Host vmk IP for iSCSI target>:3260
iscsid: session 7 connected local port 43708 to <Host vmk IP for iSCSI target>:3260
iscsid: session 7 login response status 0101
iscsid: session 7 login_rsp ret (8)
iscsid: session 7 re-opening (reopen_cnt 1)
iscsid: session 7 disconnecting conn 0x55e0293a4348, fd 22
iscsid: session 7 connecting to <Another Host vmk IP for iSCSI target>:3260
iscsid: session 7 connected local port 46362 to <Another Host vmk IP for iSCSI target>:3260
iscsid: session 7 login response status 0000
iscsid: connection7:0 is operational after recovery (1 attempts)

• Other VM iSCSI initiators do not lose connection to vSAN iSCSI targets
  
  
• vSAN Skyline Health reports no warnings  

Environment

VMware vSAN 8.x

Cause

On the impacted VMs, the iSCSI initator fails to respond to a NOP-OUT PDU from the vSAN iSCSI target within the timeout period. As a result the iSCSI target closes the connection.


In this case, ESXI /var/log/vmkernel.log will report an error similar to:

vmkwarning: cpu31:2103576)WARNING: <VM guest IP> (<VM guest iSCSI initiastor iqn): no ping reply (NOP-Out) after 5 seconds; dropping connection

Resolution

• Resolve any network connectivity issues between the VM and the iSCSI target.
  
  
• Idenify and remediate any guest/VM level operations (e.g. VM backup) which may temporarily freeze the VM/guest and prevent the iSCSI initiator from replying to the NOP-OUT PDU.