ESXi Host Unresponsiveness and VM Inaccessibility Due to Storage Latency or Fabric Issues
search cancel

ESXi Host Unresponsiveness and VM Inaccessibility Due to Storage Latency or Fabric Issues

book

Article ID: 392616

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms

 

  • The ESXi host appears connected in the vCenter Server inventory but does not respond to management operations.

  • All virtual machines (VMs) on the affected host become inaccessible.

  • Power operations such as Power On, Power Off, or Reset fail for VMs residing on the host.

  • Restarting ESXi management agents (hostd, vpxa) does not restore host or VM responsiveness.

 

Environment

VMware ESX 7.x
VMware ESX 8.x

Cause

This issue typically occurs when high system latency or storage-related delays impact the responsiveness of the ESXi management service, hostd. As a result, the host becomes unresponsive to management operations while still appearing connected in vCenter. Contributing factors may include:

  • Storage array performance degradation

  • Fabric issues, such as SAN switch/zoning delays or intermittent path failures

  • SCSI command failures with sense key 0xB / ASC 44/00 indicating Internal Target Failure

  • Aborted commands observed due to path or array-level issues

Log Messages Observed:

  • In the /var/log/vmkernel.log file, the following warning messages may be seen:

    • "ALERT: hostd performance has degraded due to high system latency"

    • "Devices/volumes experiencing 'Internal Target Failure'"

YYYY-MM-DDTHH:MM:SSZ cpu77:2101555)ALERT: hostd performance has degraded due to high system latency
YYYY-MM-DDTHH:MM:SSZ cpu69:2101555)ALERT: hostd performance has degraded due to high system latency

YYYY-MM-DDTHH:MM:SSZ cpu84:2098465)ScsiDeviceIO: 4115: Cmd(0x45b9e4484008) 0x2a, CmdSN 0x800e0032 from world 2114460 to dev "naa.600####" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0
YYYY-MM-DDTHH:MM:SSZ cpu70:2098465)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x2a (0x45d9c2651988, 2103527) to dev "naa.600####" on path "vmhba0:C0:T2:L49" Failed:

  • Sense Code 0xB 44/00 = Aborted Command / Internal Target Failure

In the /var/log/vmkwarning.log, the following messages "state in doubt; requested fast path state update..." may appear:

YYYY-MM-DDTHH:MM:SSZ cpu104:2162384)WARNING: nfnic: <1>: fnic_abort_cmd: 3890: Abort for cmd tag: 0x3fc in pending state
YYYY-MM-DDTHH:MM:SSZ cpu103:2098465)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.600####" state in doubt; requested fast path state update...

Resolution

Resolution

The issue may be caused by storage array performance degradation or a fabric-related issue. To resolve it:

  1. Engage the storage vendor to investigate latency at the storage array level.

  2. Check the storage fabric health, including SAN switches, zoning, and connectivity between the ESXi host and storage array.

  3. Monitor storage response times to identify anomalies or bottlenecks in the data path.

Workaround

To temporarily recover from the unresponsive state and regain access to the affected virtual machines (VMs), perform the following:

  1. Hard reset the affected ESXi host using the KVM/IPMI console.

  2. Upon reboot, the High Availability (HA) mechanism will trigger, causing VMs to restart on available hosts within the cluster.

Additional Information

 

Powered by Wolken Software