Error: Bootbank cannot be found at path '/bootbank'

Products

• VMware vSphere ESXi

Issue/Introduction

Introduction:

• ESXi Host reports "Bootbank cannot be found at path '/bootbank'" alerts in /var/run/log/vmkernel.log and the same can be seen as a warning on vSphere Client:

  

• Host reports no connection with the local storage, and unable to access the logs via SSH

• The logs may be accessible, but the SSH would be very slow.

• ESXi Host is marked as "Not Responding" on vSphere Client.

• Virtual Machines on the host may continue to run with no issues.

• /var/run/log/hostd.log of the ESXi host reports that the bootbank is not found:

  Hostd: --> Bootbank cannot be found at path '/bootbank'

• /var/run/log/syslog.log of the ESXi syslog reports boot device hardware failure:

  Wa(12) smartd[2104025]: t10.NVMe__HPE_NS204i2Dp_Gen102B_Boot_Controller___0100000432435000: REALLOCATED SECTOR CT below threshold (0 < 90)

• In /var/run/log/vmkernel.log SCSI errors for write failures and also NO_CONNECT (H:0x1) can be seen:

  vmkernel: cpu57:2097355)ScsiDeviceIO: 4154:Cmd(0x45bd4a129908) 0x2a, cmdId.initiator=0x430ac151f040 CmdSN 0x10e9755 from world 2097806 to dev t10.####____########_#####_____###### failed H:0x5 D:0x0 P:0x0
In(182) vmkernel: cpu38:####### HPP: HppScsiThrottleLogForDevice:523: Error status H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0 from device t10.####_____######_####______________________################## repeated 20 times, hppA$

• Run below command from SSH or Shell prompt to check the status of the boot disk

  vmkfstools -Ph -v10 bootbank

  
  Could not open bootbank
Error: Unable to access device, please check your connection to the device.

  OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

Environment

• vSphere ESXi 7.x

• vSphere ESXi 8.x

Cause

A loss of connectivity occurs between the ESXi hypervisor and the physical boot device (SD Card, USB, or local HBA)

Resolution

Fixed in ESXi 8.0 Update 3 and higher. See Download Broadcom products and software for steps to download the latest release.

To restore management and identify the hardware failure:

1. 1. Coordinate a maintenance window for the affected host.
   2. If the host is in an HA-enabled cluster, evacuate VMs if possible; otherwise, ensure HA is prepared to restart VMs on other nodes.
   3. Perform a graceful reboot of the host.
   4. Once the host is back online, immediately collect a VM-support log bundle.
   5. Engage the hardware vendor to investigate the boot device or HBA for write failures, stale cache data, or physical disconnections.
   6. If the host fails to power on or mount /bootbank after reboot, replace the faulty hardware component.

If further assistance is required, see Contact Support. Scroll to the bottom of the page and click on your respective region.

 

Additional Information

• If the logs are missing during the time of issue, check whether the scratch partition and bootbank is configured on the same device.
• If configured on same device and do not see any log records for the duration, configure scratch partition on separate device:

Creating a persistent scratch location for ESXi 8.x/7.x/6.x

Japanese KB: Bootbank がパス '/bootbank' に見つかりません