iSCSI NOP Out Timeout Leading to TCP Socket Closure

Products

• VMware vSAN

Issue/Introduction

In a vSAN iSCSI target environment, some intermittent connection drops may be observed between a Linux initiator and an ESXi target. The following symptoms are typical:

• The Linux initiator logs ISCSI_ERR_NOP_TIMEDOUT followed by a TCP connection closure.
  

  iscsid: Kernel reported iSCSI connection 0:0 error (1022 - ISCSI_ERR_NOP_TIMEDOUT: A NOP has timed out)
  iscsid: Kernel reported iSCSI connection 0:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed)
  iscsid: connection 0:0 is operational after recovery (1 attempts)

• The ESXi host (Target) logs failed to send PDU, error 32 and drops the connection.
  

  WARNING: icl_conn_send_pdus: failed to send PDU, error 32; dropping connection
  WARNING: <iqn.initiator.name>: connection error; dropping connection
  cfiscsi_ioctl_handoff: new connection from <iqn.initiator.name>...

• Disconnect duration is typically short
• Application I/O may remain unaffected

Environment

• VMware vSAN: 8.x, 9.x
• Features: vSAN iSCSI Target Service enabled
• Initiator: Physical or virtual Linux servers as iSCSI initiator
• Network: iSCSI traffic over dedicated or shared physical uplinks

Cause

The issue is caused by a NOP-Out (No Operation) timeout.

• Mechanism: The iSCSI Initiator (Linux) sends a NOP-Out packet as a protocol-level heartbeat to the Target (ESXi) to verify connectivity.
• Failure: If the Initiator does not receive a NOP-In response from the Target within the expected timeframe, it assumes the TCP transport is unresponsive and terminates the socket to initiate recovery.
• Network Latency: This is often caused by transient network congestion or packet loss affecting heartbeat packets, even if standard data I/O continues to flow.

Resolution

If the disconnects are infrequent and do not impact application stability, they may be considered transient. However, to address or investigate the root cause:

1. Network Packet Capture: Perform simultaneous packet captures on the Linux Initiator, the physical network switches, and the ESXi Target to identify where the NOP packets are being delayed or dropped.
2. Validate Multipath Settings: Ensure the Linux multipath.conf and iSCSI settings (e.g., replacement_timeout) are tuned according to vendor best practices to handle transient path failures.
3. Monitor Frequency: Correlate the timing of the timeouts with network maintenance windows or high-utilization periods on the physical uplink.
4. Hardware Check: Inspect physical cables and SFP modules for CRC errors or link flaps that could cause micro-bursts of packet loss.

Additional Information

• NOP-Out: Heartbeat sent by Initiator to Target.
• NOP-In: Response sent by Target to Initiator.
• Error 32 (Broken Pipe): Indicates the ESXi host tried to send data over a socket already closed by the remote peer (Linux).