ESXi Host Lockups During Log Collections or Storage Rescans Due to Underlying Fibre Channel Instability

Products

• VMware vSphere ESXi

Issue/Introduction

Customers may observe severe ESXi host unresponsiveness or "lockups" when performing routine administrative or troubleshooting tasks. Specific manifestations include:

• Host Hangs During Log Collection: The host becomes completely unresponsive when running vm-support or generating a diagnostic bundle. The process may stall at storage-related collection stages.
• Lockups During Storage Rescans: Initiating a "Rescan Adapters" or "Rescan Storage" from vCenter or the CLI causes the host management agents (hostd/vpxa) to hang, often requiring a hard reboot.
• Management Plane Unresponsiveness: vCenter reports the host as "Not Responding," and SSH sessions may hang specifically when executing storage-related commands (e.g., esxcli storage ... or localcli ...).
• Guest OS Impact: While the host management plane is locked, VMs may experience disk I/O timeouts, "Read-Only" file systems, or become completely frozen.

Environment

ESXi 8.x

Cause

These host lockups are typically secondary manifestations of a pre-existing, critical instability in the storage layer—most commonly Fibre Channel (FC) frame drops or SCSI H:0x2 (Host Busy) conditions.

When the underlying storage fabric is dropping frames, the HBA driver is unable to successfully issue or complete SCSI commands. This creates a backlog of pending I/O and "Device Busy" states. When a user initiates a log collection or a storage rescan, the host attempts to query every path and device for status.

In a degraded environment:

1. The rescan/log-collect triggers an influx of management-level SCSI commands.
2. Because the fabric is unstable (H:0x2), these commands do not complete and instead wait for timeouts.
3. The management agents (hostd) exhaust their available worker threads waiting for these storage responses, leading to a total lockup of the host's management plane.

Identification

To confirm if the lockup is caused by underlying storage instability, check the vmkernel.log (if accessible) or post-reboot logs for the following indicators prior to the hang:

• SCSI Status H:0x2: ScsiDeviceIO: ... failed H:0x2 D:0x0 P:0x0
• Frame Drop Events: WARNING: iodm: vmk_IodmEvent: ... FRAME DROP event has been observed
• Driver Errors: qlnativefc: ... Dropped frame(s) detected or lpfc: ... FRAME: drops=###

Resolution

The host lockups will continue to occur during administrative tasks until the primary storage fabric issue is resolved.

1. Stabilize the Fabric:

• Engage SAN/Storage vendors to investigate the physical layer (Cables, SFPs, Switch Ports) for CRC errors or frame drops.
• Ensure HBA firmware and drivers are aligned with the https://compatibilityguide.broadcom.com

2. Avoid Intensive Tasks During Instability:

• Refrain from performing storage rescans or full log collections until the fabric errors (H:0x2) are no longer observed in the logs.

3. Path Isolation:

• If the instability is isolated to a specific HBA or switch port, disable those paths to prevent the host from attempting to communicate across the degraded fabric.

4. Host Recovery:

• If a host is currently locked, a restart of management agents (/etc/init.d/hostd restart) may not be sufficient if the kernel is waiting on blocked I/O. A physical reboot of the ESXi host is often required to clear the pending command queue.