"Bootbank cannot be found at path '/bootbank'" alert happens when it runs out of AIO handles.

Products

• VMware vSphere ESXi

Issue/Introduction

When multiple virtual machines with vGPU attached are running concurrently on a single host, the following alert is logged in the vmkernel.log file:

ALERT: Bootbank cannot be found at path '/bootbank'

The following messages are also logged in the vmkernel.log file:

WARNING: FDS: ###: Could not initialize AIO handles ########: No free handles

 

 

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware ESX 9.0

Cause

ESXi can allocate up to 32,768 AIO handles for FDS on a system. When running numerous virtual machines with vGPU attached on a single host, the available AIO handles may become exhausted, resulting in an inability to open device files due to the lack of free AIO handles. When the ESXi backup.sh script attempts to execute stat() at '/bootbank', it fails due to AIO handle exhaustion, and an alert is subsequently issued.

This is a variant issue of KB 408376.

 

Resolution

Broadcom engineering team is working on the fix.

Workaround:
The issue can be avoided by either powering off the VMs that are consuming a large number of AIO handles or migrating them to a host with lower consumption.
Please refer to Resolution of KB 408376 for the detailed steps.