ESXi hosts become unresponsive during NFC operations when IO Filters are being used.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
ESXi hosts become unresponsive with hostd failing or hanging when NFC operations such as backup or replication jobs are executed on disks with an IO filter attached.

There may be messages in /var/run/log/hostd.log indicating IoTracker hostd threads taking increasingly long amounts of time to complete:

2022-06-23T09:59:46.880Z warning hostd[3044912] [Originator@6876 sub=IoTracker] In thread 3049831, preadv("/vmfs/devices/vdfm/1322229d-vdfm") took over 19214 sec.

2022-06-23T09:59:56.883Z warning hostd[3045356] [Originator@6876 sub=IoTracker] In thread 3049832, preadv("/vmfs/devices/vdfm/1322229d-vdfm") took over 19224 sec.
2022-06-23T09:59:56.883Z warning hostd[3045356] [Originator@6876 sub=IoTracker] In thread 3049833, preadv("/vmfs/devices/vdfm/1322229d-vdfm") took over 19224 sec.
2022-06-23T09:59:56.883Z warning hostd[3045356] [Originator@6876 sub=IoTracker] In thread 3049831, preadv("/vmfs/devices/vdfm/1322229d-vdfm") took over 19224 sec.
 

Environment

VMware vSphere ESXi 7.x

Cause

This issue is caused by a hostd worker thread limit exhaustion during specific NFC operations.

Resolution

This issue is resolved in vSphere ESXi 7.0 U3l (build number 21424296).

Workaround:
You can work around this issue by limiting NFCs ability to overwhelm hostd. Set the maximum asynchronous NFC threads to 2 in the hostd configuration.


1. Export the hostd configuration settings from ConfigStore to a json file using the following command.
 

configstorecli config current get -c esx -g services -k hostd -outfile tmp.json

2. Edit the tmp.json file:
 

vi tmp.json

3. Add the "max_async_threads": 2  line to the "nfcsvc"  section as seen below and save the file (Note that the other options may differ in your environment).

"nfcsvc": {
         "log_level": "INFO",
         "max_memory": 100663296,
         "max_stream_memory": 35651584,
         "max_async_threads": 2

4. Run the following command to apply the file to the ConfigStore database:
 

configstorecli config current set -c esx -g services -k hostd -infile tmp.json

5. Run the following command to restart hostd service:
 

/etc/init.d/hostd restart