Intermittent ESXi host unresponsiveness with "bootbank cannot be found" errors on HPE servers

Products

• VMware vSphere ESXi

Issue/Introduction

• The host is marked as "Not Responding" in vCenter Server, the Virtual Machines keep running on the host.
• The DCUI console displays the error: bootbank cannot be found at path "/bootbank".
• IPMI logs on the iLO report: The iLO health monitoring status of the device/adapter located in Slot ## is not responsive.
  
• The device in Slot ## is identified as the NVMe boot device.
• Attempts to access logs on the ESXi host makes the SSH session hung 

Cause

The device in Slot ## mentioned in the issue corresponds to the NVMe boot device.

This issue is caused by a firmware bug affecting the HPE NVMe boot device, specific to HPE Gen 10 or higher servers with the Boot Controller NS204i.

The ESXi kernel loses connection to the boot partition, the bootbank becomes inaccessible, causing management services to hang.

Resolution

Apply the latest firmware for the HPE NVMe SSD.

Engage HPE support to identify the firmware version compatible with the device, refer Advisory: (Revision) HPE Intel Gen12 and HPE ProLiant AMD Gen11 EPYC 9xx5 2P Servers - NS204i-u Device Minimum Firmware Requirement and Downgrade Restriction on Configurations Supporting the Device Management Communication Framework Change With HPE iLO