vpxa crashes due to "failed to open /dev/urandom: Too many open files" when it runs out of AIO handles.

Products

• VMware vSphere ESXi

Issue/Introduction

vpxa crashes with the following error:

RandomBytesPosix: failed to open /dev/urandom: Too many open files

"Not Responding" alert associated with the ESX host may be logged in vSphere Client at the time.

 

The following warning messages are also logged in the vmkernel.log file:

WARNING: FDS: ###: Could not initialize AIO handles ########: No free handles

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware ESX 9.0

Cause

ESXi can allocate up to 32,768 AIO handles for FDS on a system. When running numerous virtual machines with vGPU attached on a single host, the available AIO handles may become exhausted, resulting in an inability to open device files due to the lack of free AIO handles. Under this conditions, vpxa service may crash when attempting to open a device file.

This is a variant issue of KB 408376.

Resolution

Broadcom engineering team is working on the fix.

Workaround:

The issue can be avoided by either powering off the VMs that are consuming a large number of AIO handles or migrating them to a host with lower consumption.
Please refer to Resolution of KB 408376.