Troubleshooting intermittent SCSI command aborts and iSCSI connection resets on ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Error Messages:

• Reset to device, \Device\RaidPort0, was issued. (Guest OS Windows Event Log)
• ScsiDeviceIO: 4619: Cmd(...) failed H:0x2 D:0x0 P:0x0 (VMkernel logs)
• iscsivmk_ConnReceiveAtomic:474: ... Failed to receive data: Connection closed by peer
• NetPort: 708: Failed to acquire port non-exclusive lock

Description: Virtual machines may intermittently pause or experience I/O latency. Guest operating systems report SCSI bus resets (e.g., LSI_SAS "Reset to device"). Analysis of VMkernel logs reveals concurrent iSCSI session disconnects, recurring BUS_BUSY (Host Status 0x2) errors, and network port lock acquisition failures, indicating underlying transport instability rather than specific driver-level device faults.

Environment

VMware vSphere ESXi 8.X

Cause

• The symptoms are triggered by external transport instability causing TCP session resets. When the storage array or network fabric terminates the iSCSI TCP connection (evidenced by "Connection closed by peer"), the ESXi host cannot transmit I/O commands, resulting in BUS_BUSY errors.
• Outdated network driver and firmware combinations (specifically legacy bnxtnet driver versions) can exacerbate this by failing to handle port lock contention and TCP offload tasks effectively during periods of packet loss or network congestion, leading to the observed NetPort lock failures.

Resolution

To resolve this issue, perform the following validation and updates:

1. Verify Physical Network Infrastructure:

   • Consult your network/switch vendor documentation.
   • Check switch logs for packet drops, CRC errors, or flow control mismatches on the VLANs assigned to your iSCSI vmkernel ports.
   • Ensure MTU settings are consistent across the entire path (End-to-End) for Jumbo Frame configurations.

2. Verify Storage Array Health:

   • Engage your storage array vendor to inspect controller logs.
   • Specifically, investigate reasons for TCP FIN/RST packets or "Connection closed by peer" events on the storage target.

3. Update Network Driver and Firmware:

   • Take a snapshot/backup of the host configuration or ensure the host is in maintenance mode.
   • Review the Broadcom Compatibility Guide (Hardware Compatibility List - HCL) to identify the latest supported driver and firmware for your physical NICs (e.g., Broadcom bnxtnet).
   • Update the driver and firmware to the recommended version.

Additional Information

• Interpreting SCSI sense codes in VMware ESXi
• For further investigation of iSCSI session drops, refer to vmkernel.log filtering for iscsivmk events.