Error: Bootbank cannot be found at path '/bootbank' on ESXi host

Products

VMware vSphere ESXi

Issue/Introduction

The ESXi Host reports "Bootbank cannot be found at path '/bootbank'" alerts in /var/run/log/vmkernel.log, and the same can be seen as a warning on the vSphere Client:
```
####-##-##T##:##:##.###Z cpu## :########)Bootbank cannot be found at path ' /bootbank'
```
The host reports no connection with the local storage and is unable to access the logs via SSH.
The logs may be accessible, but the SSH connection would be very slow.
The ESXi Host is marked as "Not Responding" on the vSphere Client.
Virtual Machines on the host may continue to run with no issues.
The /var/run/log/hostd.log file of the ESXi host reports that the bootbank is not found:
```
Hostd: --> Bootbank cannot be found at path '/bootbank'
```

The /var/run/log/syslog.log file of the ESXi syslog reports a boot device hardware failure:

Wa(12) smartd[2104025]: t10.NVMe__HPE_NS204i2Dp_Gen102B_Boot_Controller___0100000432435000: REALLOCATED SECTOR CT below threshold (0 < 90)

In /var/run/log/vmkernel.log, SCSI errors for write failures and NO_CONNECT (H:0x1) can be seen:

vmkernel: cpu57:2097355)ScsiDeviceIO: 4154:Cmd(0x45bd4a129908) 0x2a, cmdId.initiator=0x430ac151f040 CmdSN 0x10e9755 from world 2097806 to dev t10.####____########_#####_____###### failed H:0x5 D:0x0 P:0x0
In(182) vmkernel: cpu38:####### HPP: HppScsiThrottleLogForDevice:523: Error status H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0 from device t10.####_____######_####______________________################## repeated 20 times, hppA$

Run the following command from an SSH or Shell prompt to check the status of the boot disk:

#vmkfstools -Ph -v10 bootbank
Could not open bootbank
Error: Unable to access device, please check your connection to the device.

OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

Environment

vSphere ESXi

Cause

A loss of connectivity occurs between the ESXi hypervisor and the physical boot device (SD Card, USB, or local HBA). This is often due to hardware degradation or firmware issues on controllers such as the Cisco UCS M7 series or the HPE NS204i.

Resolution

Fixed in ESXi 8.0 Update 3 and higher. See Download Broadcom products and software for steps to download the latest release.

Follow these steps to restore management and investigate the hardware:

Back up the ESXi configuration immediately using How to back up and restore the ESXi host configuration. A reboot may result in total boot failure if the device has failed permanently.
Coordinate a maintenance window and evacuate all VMs from the host.
Perform a graceful reboot of the host.
Once the host is back online, collect a VM-support log bundle.
Engage the hardware vendor (e.g., Cisco or HPE) to inspect the boot media or HBA for hardware failures or required BIOS/firmware updates.
If the issue persists, replace the faulty boot media.

For defects and enhancements, subscribe to this article (see Subscribe to a Broadcom knowledge article by article or product ) to receive updates on fix status.

If further assistance is required, see Contact Support. Scroll to the bottom of the page and click on your respective region.

Additional Information

If the logs are missing during the time of issue, check whether the scratch partition and bootbank is configured on the same device.
If configured on same device and do not see any log records for the duration, configure scratch partition on separate device:

Creating a persistent scratch location for ESXi 8.x/7.x/6.x

Japanese KB: Bootbank がパス '/bootbank' に見つかりません

ESXi host disconnected from vCenter and unresponsive Virtual Machines - Marvell 88SE9230 AHCI