Bootbank cannot be found at path '/bootbank' errors being seen after upgrading to ESXi 7.0 U2

Products

• VMware vSphere ESXi

Issue/Introduction

To resolve the bootbank cannot be found.

Symptoms:

• ESXi host boots off of USB SD card.
• ESXi goes into  non-responding status.
• The following vCenter alarm is generated 

Alarm 'Host error' on esxi.example.com triggered by event 75855264 'Issue detected on esxi.example.com in DATACENTER: Bootbank cannot be found at path '/bootbank'

• vmkwarning.log may show entries similar to:

YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...

• vmkernel.log  may show entries similar to:

YYYY-MM-DDTHH:MM:SS.Z cpuxx:7579038)VMW_SATP_LOCAL: satp_local_updatePath:856: Failed to update path "vmhba32:C0:T0:L0" state. Status=Transient storage condition, suggest retry
YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097681)ScsiPath: 8058: Cancelled Cmd(0x45c900f21240) 0x1a, cmdId.initiator=0x4538f041a598 CmdSN 0x1407e8 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.
YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097681)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "mpx.vmhba32:C0:T0:L0" state in doubt; requested fast path state update...
YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097681)ScsiDeviceIO: 4315: Cmd(0x45c900f21240) 0x1a, cmdId.initiator=0x4538f041a598 CmdSN 0x1407e8 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x5 D:0x0 P:0x0 Cancelled from path layer. Cmd count Active:1
YYYY-MM-DDTHH:MM:SS.Z cpuxx:2097682)ScsiPath: 8058: Cancelled Cmd(0x45c900e60040) 0x0, cmdId.initiator=0x4538e9b1bc58 CmdSN 0x0 from world 0 to path "vmhba32:C0:T0:L0". Cmd count Active:0 Queued:2.


NOTE: The preceding log excerpts are only examples. Date, time and environmental variables may vary depending on your environment

Environment

VMware vSphere ESXi 7.0.x

Cause

USB devices have a small queue depth and due to a race condition in the ESXi storage stack, some I/O operations might not get to the device. Such I/Os queue in the ESXi storage stack and ultimately time out.

Resolution

This issue is resolved in VMware vSphere ESXi 7.0 U2c and later. To download go to the Broadcom Downloads page.

To workaround this issue, delay the storage-path-claim service to allow ESXi to retrieve the correct bootbank image from its storage device. Please refer the below KB:

https://knowledge.broadcom.com/external/article?legacyId=2149444

 

Additional Information

Impact/Risks:
Reboot of ESXi host.