ESXi host goes to a not responding state due to hostd service being blocked by VMFS datastore I/O operations

Products

• VMware vSphere ESX 7.x

Issue/Introduction

• The vSphere Client shows the ESXi host as Not Responding.
• No storage I/O failures or latency increases were observed.
• The file /var/run/log/hostd.log shows entries similar to the following:

YYYY-MM-DDTHH:MM:SS.###Z warning hostd[<pid>] [Originator@6876 sub=IoTracker] In thread <thread id>, stat("/vmfs/volumes/########-########-####-############/path") took over ###sec.
YYYY-MM-DDTHH:MM:SS.###Z warning hostd[<pid>] [Originator@6876 sub=IoTracker] In thread <thread id>, open("/vmfs/volumes/########-########-####-############/path") took over ### sec.
YYYY-MM-DDTHH:MM:SS.###Z warning hostd[<pid>] [Originator@6876 sub=IoTracker] In thread <thread id>, access("/vmfs/volumes/########-########-####-############/path") took over ### sec.

Note: The above log entries may appear not only in this issue but also due to other causes.

Environment

• VMware vSphere ESXi 7.0.0
• VMware vSphere ESXi 7.0.1
• VMware vSphere ESXi 7.0.2

Cause

There is a known issue related to VMFS mutex locks that can cause hostd threads to become stuck during I/O operations.

To verify if this issue matches, please collect live dumps of both hostd and vmkernel, and contact Broadcom VCF Support.

For instructions on how to obtain live dumps, please refer to the following KB article:
Generating Live core dump for ESXi host

Resolution

This issue is fixed in VMware vSphere ESXi 7.0 Update 3 or later.

Restarting the ESXi host is required for recovery.