SocketTimeoutException during Storage HBA rescan due to Fibre Channel frame drops, ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• Rescanning Storage HBA hangs for exactly 10 minutes.
• The vCenter Server reports the host status as Not Responding during the rescan.
• vobd.all logs demonstrate path redundancy degradation (esx.problem.storage.redundancy.degraded).
• The following error appears in vCenter: invocation failed with "java.net.SocketTimeoutException: 600,000 milliseconds timeout on connection http-outgoing-#### [ACTIVE]"
• Logs show FRAME DROP events in vmkernel.log on a specific vmhba (e.g., vmhba3): WARNING: iodm: vmk_IodmEvent:191: vmhba3: FRAME DROP event has been observed

Environment

• VMware vSphere ESXi 8.x
• Fibre Channel (FC) SAN infrastructure
• Brocade SAN Switches

Cause

Physical SAN fabric degradation causes this issue, typically a failing Small Form-Factor Pluggable (SFP) transceiver or a faulty cable on the switch port. The faulty component induces continuous Fibre Channel frame drops and SCSI command aborts. When a rescan is initiated, the kernel SCSI subsystem attempts to poll dead or unacknowledged paths, which exhausts hostd worker threads and results in a 10-minute timeout.

Resolution

1. Identify the faulty physical link by checking vmkernel.log for frame drop warnings isolated to a specific vmhba using the following command: grep Dropped /var/log/vmkernel.log
2. Decommission and replace the degraded SFP transceiver or cable on the upstream SAN switch port identified in Step 1.
3. Verify port stability by monitoring switch port error counters (CRC errors, Loss of Sync) to ensure zero increments.
4. Perform a clean cluster-wide storage HBA rescan to clear residual dead paths and confirm that Device Average Response Time (DAVG) returns to normal baselines.

Subscribe to this article to be updated on any changes to the status of this issue: How to Subscribe to a Knowledge Base Article.

Additional Information

• High latency observed to FC channel datastores due to dropped frames
• ESXi storage rescan task hangs after the storage device is unmasked
• To speak with a customer representative or a Support Engineer see Contact Support. Scroll to the bottom of the page and click on your respective region.