ESXi Host iSCSI Connection Drops Error 1011 to HPE 3PAR Alletra MP

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts experience recurring iSCSI connection drops to the HPE 3PAR/Alletra MP storage array. These connection drops manifest as severe I/O stalls and host agent exhaustion. The issue isolates strictly to the 3PAR targets.

The vmkernel.log reports the following errors:

iscsi_vmk: iscsivmk_StopConnection:736: vmhba64:CH:0 T:42 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
Failed to receive data: Connection closed by peer

Additionally, syslog.log will indicate NOP timeouts (Event:4) and connection resets by the peer (Event:6), resulting in Error 1011 (iSCSI connection timeout) while the session is in State 3 (LOGGED_IN).

Environment

VMware ESXi
HPE 3PAR / Alletra MP

Cause

The issue can occur when a single storage controller node handles a disproportionate amount of the I/O workload compared to its peer, causing the array's iSCSI service on the overloaded node to become intermittently unresponsive.

Resolution

Since this issue isolates to the third-party storage controller's handling of the iSCSI service under heavy load, Broadcom recommends opening a support case directly with HPE for investigation.

As a potential mitigation based on observations from similar environments, we recommend analyzing and comparing I/O load across the LUN Active/Optimized (AO) ownership across the storage nodes. Evenly distributing the I/O payload between the nodes helps prevent a single controller from hitting this issue.

Additional Information

For further details on third-party vendor engagement, refer to your HPE support entitlements.