Alert hostd performance has degraded due to high system latency

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Some hosts going into not responding state causing the VM's to become inaccessible.
  
  
• An alarm is triggered in the VC from multiple hosts: 

Alarm 'Host error' on esxi host triggered by event 14803934 'Issue detected on hostname.domain.com in Datacenter: hostd performance has degraded due to high system latency'.

In the /var/run/log/vmkernel.log files on the ESXi host 6.x/7.x/8.x, you see similar warning messages:

<YYYY-MM-DD>T<time>Z In (182) vmkernel: cpu29:2097327)ScsiDeviceIO: 4656: Cmd(0x45bb42e5b880) 0xfe, cmdId.initiator=0x430a74fe9980 CmdSN 0x18a9db from world 2099102 to dev "naa.xxxxxxxxxxxxxxxxxxx" failed H:0x5 D:0x0 P:0x0. Cmd count Active:0

 

<YYYY-MM-DD>T<time>Z In (182) vmkernel: cpu7:2097305)ScsiDeviceIO: 4672: Cmd(0x45bb42e79280) 0xfe, CmdSN 0x18a9cf from world 2099102 to dev "naa.xxxxxxxxxxxxxxxxxxx" failed H:0x3 D:0x0 P:0x0

 

• Unable to check logs or run any esxli commands due to sluggish performance of the host.

Environment

VMware ESX 6.x

VMware ESX 7.x

VMware ESX 8.x

 

Cause

Host logs reveal that one or more storage LUNs are experiencing timeouts for commands in-flight to the storage array. This typically indicates an issue with the storage array, communication paths, or underlying infrastructure, potentially leading to degraded performance or connectivity disruptions.

Host Status- [0x3] -TIME_OUT --> This status is returned when the command in-flight to the array times out.

Resolution

In this situation: Reboot the host to get access to all the VMs which were inaccessible due to ESXi sluggish behavior.

Engage the storage/Switch vendor to investigate the root cause of the command timeouts observed in the host logs.