ESXi host unresponsive to management services during iSCSI storage maintenance or firmware upgrades

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi host becomes unresponsive to vCenter management commands and direct console access (hostd/vpxa lockup) during scheduled storage maintenance, such as rolling firmware upgrades on a Pure Storage array. Affected VMs may lose network connectivity, and restarting management agents (services.sh) fails to restore the host state.

• Host status changes to 'Not Responding' in vCenter Server.
• Multiple iSCSI session recovery timeouts observed in vobd.log: [vob.iscsi.session.recovery.timeout] iScsi session recovery timeout for vmhba64:C1:T0.
• Peer-initiated disconnects recorded in vmkwarning.log: Failed to receive data: Connection closed by peer.
• Massive I/O queuing and task management aborts in vmkernel.log.
• Datastore inaccessibility (APD) alerts for storage volumes.

 

 

Environment

 

• VMware ESXi 7.x / 8.x
• iSCSI Storage Arrays (specifically Pure Storage during Storage Processor reboots)

 

Cause

This issue occurs when an environmental mismatch prevents the ESXi host from successfully failing over active I/O to standby paths during a target reboot. Common causes include:

1. Delayed ACK: Enabled by default in ESXi, which can delay path failover during congestion or TCP resets.
2. Login Timeout: A default value (5 seconds) that is insufficient for certain arrays to re-establish paths during a controller failover.
3. Stale Sessions: 'Ghost' iSCSI sessions that do not tear down correctly, causing the initiator to wait indefinitely.

Resolution

To prevent host lockups during future storage maintenance, perform the following sequential steps:

1. Verify that Delayed ACK is globally disabled on all iSCSI software adapters.
2. Set the Login Timeout to 30 seconds for iSCSI initiators connected to Pure Storage arrays.
3. Audit physical NIC firmware and drivers to ensure they match the cluster baseline and Broadcom Compatibility Guide.
4. Engage the storage vendor to verify array logs (SCSI sense codes) to ensure proper TCP reset behavior was initiated during the failover.