Virtual Machines Unresponsive Due to Storage Latency and Hung vMotion Tasks on VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines (VMs) residing on a specific VMware vSphere ESXi host become unresponsive, and active storage vMotion tasks may hang indefinitely, failing to progress or cancel via the user interface.
• Virtual machines on the affected ESXi host do not respond to guest OS requests.
• Storage vMotion tasks are stuck at a specific percentage (e.g., 36%) and cannot be cancelled via the vCenter Server or host UI.
• The management interface may become sluggish or unresponsive due to I/O pressure on the kernel.
• esxtop displays elevated Device Average Latency (DAVG/cmd) values for affected datastores.
• System logs contain warnings regarding device performance and heartbeat timeouts.

Environment

• Product: VMware vSphere ESXi 8.x

Cause

• Critical storage latency, specifically high DAVG/cmd (Device Average Latency), results in LUN congestion or backend storage unresponsiveness.
• This latency saturates the host I/O path, causing the VMkernel to queue I/O requests.
• This kernel-level saturation prevents the successful cancellation of pending storage tasks, such as vMotion, causing them to hang and effectively locking the host management layer regarding those specific datastores.

Resolution

Perform a hard reboot of the affected ESXi host to clear the hung I/O stack and restore host communication.

Prerequisites

• Verify that HA (High Availability) is enabled and functioning in the cluster to allow virtual machines to fail over to remaining hosts.
• Performing a hard reboot on an ESXi host with unresponsive storage tasks carries a high risk of data loss, virtual machine file system corruption, and stale file locks.

Steps

1. Diagnostic Confirmation: Connect to the ESXi host via SSH and run esxtop. Press d to switch to the device view and monitor DAVG/cmd values. High latency (typically >30ms) across multiple LUNs confirms the storage subsystem is the bottleneck.
2. Review Logs: Examine the vmkernel logs to confirm I/O deterioration: /var/run/log/vmkernel.log # Look for entries similar to: WARNING: ScsiDeviceIO: 1513: Device <naa_id> performance has deteriorated.
3. Hard Reboot: Execute a hard reboot of the affected host, <esxi_host_hostname>. This forces the termination of the hung storage process and resets the HBA/storage adapter state. # If the host remains responsive to CLI: esxcli system shutdown reboot -r "Storage latency hang" # If the host is unresponsive, physically power cycle the host.
4. Verification: Monitor the vSphere cluster to confirm that virtual machines successfully fail over and register on healthy hosts. # Verify VM registration status via PowerCLI or vCenter UI: Get-VM -Location <cluster_name> | Where-Object {$_.VMHost -eq "<esxi_host_hostname>"}
5. Storage Investigation: Review the physical storage fabric, SAN switches, and storage array controllers to identify the root cause of the latency spikes (DAVG).

Additional Information

• Multiple VMs experience latency issues (KB 392450)
• Using esxtop to identify storage performance issues for ESXi (KB 344099)
• If the issue persists after host hardware validation, please Contact Broadcom Support.