ESXi host Not Responding with "bootbank cannot be found" errors on HPE servers
search cancel

ESXi host Not Responding with "bootbank cannot be found" errors on HPE servers

book

Article ID: 437059

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The host is marked as "Not Responding" in vCenter Server, the Virtual Machines keep running on the host.
  • The DCUI console displays the error: Bootbank cannot be found at path '/bootbank'

  • IPMI logs on the iLO may report: The iLO health monitoring status of the device/adapter located in Slot ## is not responsive.

  • The device in Slot ## is identified as the NVMe boot device.
  • Attempts to access logs on the ESXi host makes the SSH session may hung.
  • Under /var/run/log/syslog.log, below error log snips are found:

    YYYY-MM-DDTHH:MM:SS Wa(##) smartd[#######]: t10.NVMe____########___________________________############: REALLOCATED SECTOR CT below threshold (0 < 90)

  • Under /var/run/log/vmkernel.log, below error log snips are found:

    YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu##:#####)FSS: ####: Failed to open file 'hpilo-d0####'; Requested flags 0x5, world: ####### [smadv], (Existing flags 0x0, world: 0 [unknown]): Busy

  • Under /var/run/log/vmkwarning.log, below error log snips are found:

    YYYY-MM-DDTHH:MM:SS Wa(###) vmkwarning: cpu##:#####)WARNING: PLOG: PLOGProbeDevice:####: Failed to read the device <t10.NVMe____HPE_NS204i2Du_Gen11_Boot_Controller______############:#> : Not found

    YYYY-MM-DDTHH:MM:SS Wa(###) vmkwarning: cpu##:#####)WARNING: HPP: HppNvmeThrottleLogForDevice:###: Admin NVMe Cmd 0x6 (0x#######, #) to dev "t10.NVMe____HPE_NS204i2Du_Gen11_Boot_Controller______############:#" on path "vmhba0:C0:T0:L0" Failed:

    YYYY-MM-DDTHH:MM:SS In(###) vmkernel: cpu76:######)PLOG: PLOGProbeDevice:####: Probe could not open device <t10.NVMe____HPE_NS204i2Du_Gen11_Boot_Controller______############:#> : Permission denied

Cause

  • The device in Slot ## mentioned in the issue corresponds to the NVMe boot device.
  • This issue is caused by a firmware bug affecting the HPE NVMe boot device, specific to HPE Gen 10 or higher servers with the Boot Controller NS204i.
  • The ESXi kernel loses connection to the boot partition, the bootbank becomes inaccessible, causing management services to hang.

Resolution

Follow the steps below, to resolve this issue.

Powered by Wolken Software