Alert hostd performance has degraded due to high system latency

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Some hosts going into not responding state causing the VM's to become inaccessible.
  
  
• An alarm is triggered in the VC from multiple hosts: 

Alarm 'Host error' on esxi host triggered by event 14803934 'Issue detected on hostname.domain.com in Datacenter: hostd performance has degraded due to high system latency'.

In the /var/run/log/vmkernel.log files on the ESXi host 6.x/7.x/8.x, you see similar warning messages:

<YYYY-MM-DD>T<time>Z In (182) vmkernel: cpu29:2097327)ScsiDeviceIO: 4656: Cmd(0x45bb42e5b880) 0xfe, cmdId.initiator=0x430a74fe9980 CmdSN 0x18a9db from world 2099102 to dev "naa.#######################" failed H:0x5 D:0x0 P:0x0. Cmd count Active:0

 

<YYYY-MM-DD>T<time>Z In (182) vmkernel: cpu7:2097305)ScsiDeviceIO: 4672: Cmd(0x45bb42e79280) 0xfe, CmdSN 0x18a9cf from world 2099102 to dev "naa.#######################" failed H:0x3 D:0x0 P:0x0

 

• Unable to check logs or run any esxcli commands due to sluggish performance of the host.

Environment

VMware ESX 6.x

VMware ESX 7.x

VMware ESX 8.x

 

Cause

Host logs reveal that one or more storage LUNs are experiencing timeouts for commands in-flight to the storage array. This typically indicates an issue with the storage array, communication paths, or underlying infrastructure, potentially leading to degraded performance or connectivity disruptions.

Host Status- [0x3] -TIME_OUT --> This status is returned when the command in-flight to the array times out.

Resolution

• Reboot the host to get access to all the VMs which are inaccessible because of ESXi sluggish behavior.
• Engage the storage/Switch vendor to investigate the cause of SCSI command failures observed in /var/run/log/vmkernel.log

Additional Information

• SCSI sense codes are used by storage devices to report detailed error and status information back to the initiator (OS, HBA, or hypervisor). Understanding them is essential for diagnosing disk, path, or array-level issues.
• Refer the following document for more details on SCSI Sense Codes: Interpreting SCSI sense codes in VMware ESXi
• Online utility to help decode SCSI sense codes from VMware ESXi logs into human-readable explanations: SCSI sense code decoder