ESXi host not responding and sluggish VM performance due to storage latency
search cancel

ESXi host not responding and sluggish VM performance due to storage latency

book

Article ID: 415377

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0

Issue/Introduction

  • The ESXi host enters a 'Not Responding' state in vCenter.

  • Virtual machines experience sluggish performance or become unresponsive.

  • Powering on virtual machines is significantly delayed following a host reboot.

  • The console for Linux-based VMs (including the vCenter Server VM) display file system errors.

  • The following snippets are observed in /var/run/log/vmkwarning.log on the ESXi host.

    Wa(180) vmkwarning: cpu##:####)WARNING: ScsiDeviceIO: ####: READ CAPACITY on device  "naa.####" from Plugin "HPP" failed. I/O error
Wa(180) vmkwarning: cpu##:####)WARNING: ScsiDeviceIO: ####: READ CAPACITY on device "naa.####" from Plugin "HPP" failed. I/O error
Wa(180) vmkwarning: cpu##:####)WARNING: ScsiDeviceIO: ####: READ CAPACITY on device "naa.####" from Plugin "HPP" failed. I/O error

  • The following snippets are observed in /var/run/log/vmkernel.log on the ESXi host.

    In(182) vmkernel: cpu#:####)HPP: HppScsiThrottleLogForDevice:523: Error status H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. from device naa.#### repeated 640 times, hppAction = 3]
In(182) vmkernel: cpu#:####)HPP: HppScsiThrottleLogForDevice:523: Error status H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. from device naa.#### repeated 1280 times, hppAction = 3]

Environment

  • VMware ESXi 7.x

  • VMware ESXi 8.x

Cause

There is an underlying issue with the Storage device (I/O error). The Host's storage stack continues to send and retry SCSI commands to the dead device, which subsequently exhausts the management agents (hostd and vpxa). Since the host assumes the device is still available, it retries indefinitely, leading to CPU spikes and Host becoming unresponsive.

Resolution

In some cases, the storage level issue may be resolved, but the errors persist at the ESXi level. In that case, a storage rescan on the host or a host reboot may restore access to the device or datastore.

Perform a storage rescan on the ESXi host:

  1. Identify the Storage Adapter (HBA) by locating the naaID from the log snippet and running the following command:
    esxcli storage core path list -d naaID

  2. Note the storage adapter (HBA) returned in the previous step and perform a localized storage rescan:
    esxcli storage core adapter rescan -A adaptor_name

  3. To scan all HBA's run the following command:
    ​​​esxcli storage core adapter rescan -a

Note: Replace esxcli with localcli if the hostd agent is completely unresponsive. See esxcli vs localcli commands

If the issue persists after a storage rescan and host reboot, engage the underlying storage vendor to investigate further.

Additional Information

Powered by Wolken Software