Error: "Failed to open file 'hpilo-d0ccb##" floods vmkernel logs and ESXi hosts become unresponsive

Products

• VMware vSphere ESXi

Issue/Introduction

• You see the following error repeated in the vmkernel log multiple times per second: 
  

  • YYYY-MM-DDTHH:MM:SS.483Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
    YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)ilo : Open could not dequeue a packet

• This message is repeated multiple times per second. Note: It is normal to see these messages every 30 seconds or so. However, more frequent entries can cause the storage to disconnect.
• The ESXi host may be in a hung or unresponsive state, or it may have experienced an unexpected reboot.
• VMs will continue to run in host memory but eventually they will show as invalid.
• The below error may be seen in the environment as well: 

  • 503 Service Unavailable (Failed to connect to endpoint:.. 

• The configured scratch location can be wiped and set back to the default.

Environment

• VMware vSphere 8.0.#
• VMware vSphere 7.0.#

Cause

• As per HPE documentation, HPE WBEM Provider will try to open a channel with HPE Integrated Lights-Out (iLO) every 30 seconds.
• It is a normal message that will also be seen with HPE Agentless Management Service (AMS), and HPE Smart Update Tool (iSUT) when trying to find an unused hpilo channel.
• It will be seen in all VMware ESXi versions, e.g, ESXi 6.5, 6.7 and 7.x that run the native versions of the hpilo driver.
• The message is logged by the operating system in the vmkernel.log when the ilo driver fails to open a command for a channel that is already in use. Read more at HPE Support Center.
• In this case, the frequency of logging is higher than expected i.e. excessive entries per second (3 per channel, 15 channels in total).
• ESXi - /var/log/vmkernel.log
  

  YYYY-MM-DDTHH:MM:SS.483Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)ilo : Open could not dequeue a packet
  
  YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.509Z cpu51:2108940)ilo : Open could not dequeue a packet
  
  YYYY-MM-DDTHH:MM:SS.509Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.521Z cpu51:2108940)ilo : Open could not dequeue a packet

• Note: By design, logging of 15 iLO channels numerates in chorological order. viz hpilo-d0ccb01, hpilo-d0ccb02. till hpilo-d0ccb15. The probe repeats once one cycle of all 15 channels completes.
• Generally these messages can be ignored unless it is repeating multiple times per second. This will clog up I/O causing storage disconnections and VM latency or failures. 

Resolution

1. Update HPE drivers and firmware to the latest compatible version and reboot the host.
2. For more information on this issue, it is recommended to contact HPE hardware support.

Additional Information

Japanese Version:
vmkernel ログにエラー "Failed to open file 'hpilo-d0ccbXX" が多量に出力されて ESXi が応答なしとなる