The ESXi host goes into a 'Not Responding' state, and virtual machines also experience sluggish performance or occasionally become unresponsive

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• The ESXi host goes into a 'Not Responding' state.
  
  
• Following a host reboot, powering on VMs is significantly delayed.
  
  
• The console for Linux-based VMs, including the vCenter Server VM, is displaying file system errors.
  
  
• The following snippets are observed in /var/run/log/vmkwarning.log on the ESXi host.
  YYYY-MM-DDTHH:MM:SS.348Z Wa(180) vmkwarning: cpu57:2098713)WARNING: ScsiDeviceIO: 12936: READ CAPACITY on device  "naa.###############################" from Plugin "HPP" failed. I/O error
YYYY-MM-DDTHH:MM:SS.348Z Wa(180) vmkwarning: cpu57:2098713)WARNING: ScsiDeviceIO: 12936: READ CAPACITY on device "naa.###############################" from Plugin "HPP" failed. I/O error
YYYY-MM-DDTHH:MM:SS.348Z Wa(180) vmkwarning: cpu57:2098713)WARNING: ScsiDeviceIO: 12936: READ CAPACITY on device "naa.###############################" from Plugin "HPP" failed. I/O error
  
  
• The following snippets are observed in /var/run/log/vmkernel.log on the ESXi host.
  YYYY-MM-DDTHH:MM:SS.719Z In(182) vmkernel: cpu7:2098199)HPP: HppScsiThrottleLogForDevice:523: Error status H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. from device naa.###############################ada repeated 640 times, hppAction = 3]
YYYY-MM-DDTHH:MM:SS.920Z In(182) vmkernel: cpu2:2098199)HPP: HppScsiThrottleLogForDevice:523: Error status H:0x8 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0. from device naa.###############################ada repeated 1280 times, hppAction = 3

Environment

ESXi 7.x
ESXi 8.x

Cause

There is a potential underlying issue with the storage (I/O error).

Resolution

In some cases, the storage level issue may be resolved, but the errors persists at ESXi level. In that case, a storage rescan on the host or a host reboot may restore access to the device/datastore. To perform a storage rescan on the ESXi host, follow the steps below:

• Identify the Storage Adapter (HBA). To do so, find the naaID from the log snippet and run following command.
  esxcli storage core path list -d naaID


• Note the storage adapter (HBA) identified in the output and perform a storage rescan by running following command.
  esxcli storage core adapter rescan -A adaptor_name


• To scan all HBA's run the following command
  ​​​esxcli storage core adapter rescan -a

** Replace esxcli with localcli if the hostd is unresponsive. Please refer esxcli vs localcli commands

If issues persist after a storage rescan and host reboot, engage the storage vendor to investigate further.

Additional Information

Please refer to Identifying Fibre Channel, iSCSI, and NFS storage issues on ESXi hosts.