VM and ESXi host become unresponsive with high KAVG due to local storage controller failure
search cancel

VM and ESXi host become unresponsive with high KAVG due to local storage controller failure

book

Article ID: 443119

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • A production Virtual Machine (VM) becomes "stuck" or completely unresponsive.
  • Snapshot operations or power tasks (Reboot, Shut Down) fail or hang indefinitely.
  • The ESXi host management agents (hostd/vpxa) may become slow or unresponsive via vCenter and SSH.
  • Using esxtop in the Disk Device view (u) or Disk Adapter view (d) reveals extremely high KAVG (Kernel Average Latency) values (e.g., >500ms or even several seconds) for a specific HBA (e.g., vmhba16).
  • Physical device status for the affected controller may show as "Dead" or "Error."

Environment

VMware vSphere ESXi 8.0.x, 7.0.x

Cause

The issue is caused by a hardware-level failure or connectivity loss of a local storage controller. When a controller stops responding to I/O requests, the ESXi kernel continues to queue commands to the hardware.

Because the hardware never acknowledges these commands, the kernel threads become blocked, leading to high KAVG. This "I/O wait" state effectively hangs the VM and management processes that rely on the kernel's storage stack.

Resolution

Because the ESXi management agents are hung waiting for the hardware, standard software-based restarts of hostd or vpxa are usually ineffective.

  1. Perform a Host Hard Reset: Use the server's out-of-band management interface (e.g., iDRAC, iLO, or IPMI) to perform a cold reboot or hard reset of the physical host.
  2. Hardware Validation: During the boot sequence, enter the System Setup (BIOS) or the Storage Controller's configuration utility to verify if the controller and its associated physical disks are detected.
  3. Inspect Hardware Logs: Check the hardware vendor's logs (SEL/LC logs) for specific errors related to the storage controller or PCI bus.
  4. Log Verification: Once the host is back online, review /var/run/log/vmkernel.log for SCSI "Dead" or "Path Failure" messages.
  5. Component Replacement: If the controller is not detected in the BIOS or the errors persist after a hard reset, engage your hardware vendor to replace the faulty storage controller or cables.

Additional Information

Powered by Wolken Software