Error: Bootbank cannot be found at path '/bootbank' on ESXi 7.0 U2

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi hosts booting from USB or SD card media may enter an unresponsive state in vCenter Server. This occurs when the storage stack cannot communicate with the boot device, leading to a missing bootbank path.

Symptoms:

• ESXi host boots from a USB or SD card.
• ESXi host may show as 'Not Responding' in vCenter.
• Multiple virtual machine related tasks fail on the impacted host with error "Operation not allowed"
• vCenter Server triggers the following alarm: Alarm 'Host error' on ESXi_FQDN triggered by event. Issue detected on ESXi_FQDN in datacenter: Bootbank cannot be found at path '/bootbank'
• The ESXi host logs the following error in /var/run/log/vmkernel.log:

YYYY-MM-DDTHH:MM:SS.Z cpuxx:####)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
YYYY-MM-DDTHH:MM:SS.Z cpuxx:####)ScsiPath: ####: Cancelled Cmd(0x####) 0x1a, cmdId.initiator=0x#### CmdSN 0x#### from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
YYYY-MM-DDTHH:MM:SS.Z cpuxx:####)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM:SS.Z cpuxx:####)ScsiDeviceIO: ####: Cmd(0x####) 0x1a, cmdId.initiator=0x#### CmdSN 0x1407e8 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
YYYY-MM-DDTHH:MM:SS.Z cpuxx:####)ScsiPath: ####: Cancelled Cmd(0x####) 0x0, cmdId.initiator=0x#### CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.

Environment

VMware vSphere ESXi 7.0.U2c and below.

Cause

USB devices have a small queue depth. A race condition in the ESXi storage stack causes some I/O operations to be queued rather than sent to the device, eventually leading to a timeout and loss of access to the bootbank.

Resolution

Fixed in release VMware vSphere ESXi 7.0 U2d and higher. See Download latest ISOs and patches for vSphere ESXi for steps to download this release.

Workaround

1. A host reboot is required to fully restore connectivity.
2. If an immediate upgrade is not possible, delay the storage-path-claim service so that ESXi can successfully retrieve the bootbank image. Follow the procedure outlined in KB 318029.

Additional Information

Note: It is recommended to upgrade to ESXi 8.x as vSphere 7.x has reached the End of General Support.