hostd crashes due to "Could not initialize AIO handles #########: No free handles"

Products

• VMware vSphere ESXi

Issue/Introduction

When multiple virtual machines with vGPU attached are running concurrently on a single host, the following warning message is frequently logged in the vmkernel.log file:

“WARNING: FDS: ###: Could not initialize AIO handles ########: No free handles”

Under these conditions, the following symptoms have been observed:

• Hostd process crashes
• Hardware health alarms triggered
• Failures when attempting to open device files
• Virtual machine crashes resulting in Blue Screen of Death (BSOD)

---

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware ESX 9.0

Cause

ESXi can allocate up to 32,768 AIO handles for FDS on a system. When running numerous virtual machines with vGPU attached on a single host, the available AIO handles may become exhausted, resulting in an inability to open device files due to the lack of free AIO handles.

Under these conditions, the hostd service may crash when attempting to open a device file, hardware sensors may fail to be monitored if ESXi services cannot access the IPMI device file, and other instability symptoms may be observed.

Resolution

Broadcom engineering team is working on the fix.

Workaround:
The issue can be avoided by either powering off the VMs that are consuming a large number of AIO handles or migrating them to a host with lower consumption. The amount of AIO handle usage can be confirmed using the vmkvsitools in ESXi shell as below. 

In the output of the following command line, VMs with large values in the first column are consuming a significant number of AIO handles. The second column indicates the process ID (VMX Cartel ID) of the VM.

# vmkvsitools lsof | awk '$3=="CHAR" {print $0}' | grep vmgfx | awk '$2=="vmx" {print $1}' | sort | uniq -c | sort -k 1 -n


Ex)
# vmkvsitools lsof | awk '$3=="CHAR" {print $0}' | grep vmgfx | awk '$2=="vmx" {print $1}' | sort | uniq -c | sort -k 1 -n
<Value1>  <VMX_Cartel_ID1>
<Value2>  <VMX_Cartel_ID2>
<Value3>  <VMX_Cartel_ID3>
...

The VMX Cartel ID of VM can be identified using  esxcli vm process list.

# esxcli vm process list


Ex)
# esxcli vm process list
<VM Name>
   World ID: ########
   Process ID: ###
   VMX Cartel ID: #######
   UUID: ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
   Display Name: ####-########-####-####-####-############
   Config File: ##########################################
...