Bootbank cannot be found at path '/bootbank' errors being seen after upgrading to ESXi 7.0 U2

Products

• VMware vSphere ESXi

Issue/Introduction

To resolve the bootbank cannot be found.

Symptoms:

• ESXi host boots off of USB SD card.
• ESXi goes into  non-responding status.
• The following vCenter alarm is generated 

Alarm 'Host error' on esxi.example.com triggered by event 75855264 'Issue detected on esxi.example.com in DATACENTER: Bootbank cannot be found at path '/bootbank'

• vmkwarning.log may show entries similar to:

2021-05-17T03:53:57.655Z cpu22:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-05-17T03:54:17.280Z cpu22:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...

• vmkernel.log  may show entries similar to:

2021-05-16T22:54:20.120Z cpu40:7579038)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
2021-05-16T22:54:29.999Z cpu68:2097681)ScsiPath: 8058: Cancelled Cmd(0x45c900f21240) 0x1a, cmdId.initiator=0x4538f041a598 CmdSN 0x1407e8 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
2021-05-16T22:54:29.999Z cpu68:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
2021-05-16T22:54:29.999Z cpu68:2097681)ScsiDeviceIO: 4315: Cmd(0x45c900f21240) 0x1a, cmdId.initiator=0x4538f041a598 CmdSN 0x1407e8 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
2021-05-16T22:54:31.120Z cpu42:2097682)ScsiPath: 8058: Cancelled Cmd(0x45c900e60040) 0x0, cmdId.initiator=0x4538e9b1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.


NOTE:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment

Environment

VMware vSphere ESXi 7.0.x

Cause

USB devices have a small queue depth and due to a race condition in the ESXi storage stack, some I/O operations might not get to the device. Such I/Os queue in the ESXi storage stack and ultimately time out.

Resolution

This issue is resolved in VMware vSphere ESXi 7.0 U2c and later. To download go to the Broadcom Downloads page.

Workaround:

None

Additional Information

Impact/Risks:
Reboot of ESXi host.