Bootbank cannot be found at path '/bootbank'

Products

• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

Symptoms:

• ESXi Host reports "Bootbank cannot be found at path '/bootbank'" alerts in /var/run/log/vmkernel.log and the same can be seen as a warning on vSphere Client:

  

• Host reports no connection with the local storage, and unable to access the logs via SSH. (OR)

• The logs may be accessible, but the SSH would be very slow.

• ESXi Host is marked as "Not Responding" on vSphere Client.

• VMs on the host may continue to run with no issues.

• /var/run/log/hostd.log of the ESXi host reports that the bootbank is not found:

  YYYY-MM-DDTHH:MM:SS.SSSZ Hostd: --> Bootbank cannot be found at path '/bootbank'

• In /var/run/log/vmkernel.log SCSI errors for write failures and also NO_CONNECT (H:0x1) can be seen:

  YYYY-MM-DDTHH:MM:SS.SSSZ vmkernel: cpu57:2097355)ScsiDeviceIO: 4154:Cmd(0x45bd4a129908) 0x2a, cmdId.initiator=0x430ac151f040 CmdSN 0x10e9755 from world 2097806 to dev t10.####____########_#####_____###### failed H:0x5 D:0x0 P:0x0
YYYY-MM-DDTHH:MM:SS.SSSZ In(182) vmkernel: cpu38:2097282)HPP: HppScsiThrottleLogForDevice:523: Error status H:0x1 D:0x0 P:0x0 Invalid sense data: 0x0 0x0 0x0 from device t10.####_____######_####______________________################## repeated 20 times, hppA$

Environment

VMware vSphere ESX 7.x

VMware vSphere ESX 8.x

Cause

This indicates that the connection between the boot device and the host is having issues, which can mean that either the boot device or the Storage HBA is faulty.

Resolution

To resolve this issue:

1. If the host is part of a HA enabled vSphere Cluster, in order to regain management of the VMs running on this host, the host may be rebooted. The VMs running on it will get restarted on the other available hosts, provided HA is configured properly.

2. On boot, the host may either boot up successfully or may not power ON. In either case, the hardware vendor needs to investigate further on the cause of the write failures and boot device disconnection issue.

   • A boot device replacement may be carried out if the device was found to be faulty. (OR)
   • HBA replacement may be carried out if the storage adapter was found to be faulty.

Additional Information

• If the logs are missing during the time of issue, check whether the scratch partition and bootbank is configured on the same device.
• If configured on same device and do not see any log records for the duration, configure scratch partition on separate device:

Creating a persistent scratch location for ESXi 8.x/7.x/6.x