ESXi Host PSOD error - Bootbank cannot be found at a path '/Bootbank' & stack trace reporting PsaSCSIDeviceTimeout, PsaStorDeivceTimeout errors

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi Host reports a PSOD recursively
  
  
• ESXi Host may boot up but users will be unable to login to the ESXi host via SSH, vSphere Client or Direct Console User Interface (DCUI)
  
  
• Users may not be able to switch screen to F12 (VMkernel logs) from the DCUI
  
  
• The Host runs into a PSOD and the stack trace points to the below errors,  
  • PsaSCSIDeviceTimeout
  • PsaStorDeivceTimeout
  • Bootbank cannot be found at a path '/Bootbank'
    Below is a PSOD screenshot stack trace for reference 
    

Environment

• vSphere ESXi
  Note - The issue can occur in all the versions of ESXi

Cause

Cause :- 

• The issue is caused due to a underlying Hardware failure
  • The error clearly indicates issues with the underlying Storage device, where the ESXi is installed
    NOTE - Issue could be due to a faulty RAID Configuration, RAID Controller card, the Backplane device or the Hard drive where the ESXi is installed.

Cause Justification :- 

• The stack trace shows "Bootbank cannot be found at path : /bootbank" indicating the VMkernel is unable to find the bootbank.
  Bootbank is a partition or area where the VMkernel and its essential components are stored.
  This partition contains the files necessary for ESXi to start and load the VMkernel, including drivers and other essential system components. 
  The bootbank must be available and accessible for a normal function of ESXi Operating system.
  
  
• The stack trace also indicates The VMkernel is issuing commands to nmpdevice where the ESXi is installed
• Logging into the DCUI shows the below error, more information on How to log into the DCUI
  "No Persistent Storage available for system Logs and data. ESXi is operating with limited system storage space. Logs and system data will be lost on reboot" indicating ESXi normal function getting failed
  
  

Resolution

• 3rd party hardware Vendor must be engaged to further investigate the issue
  
  
• A detailed Hardware diagnostics check with Offline Diagnostics must be performed to investigate the Hardware failure 

Additional Information

• If the ESXi is installed on an SD card / USB device, then check the "Guidelines for vSphere boot device" section in the KB Link 

• HPE Hardware Diagnostics

• Dell Hardware Diagnostics 

• IBM Hardware Diagnostics