Error: "Failed to open file 'hpilo-d0ccbXX" floods vmkernel logs and ESXi hosts become unresponsive

Products

• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 7.0

Issue/Introduction

• You see the following error repeated in the vmkernel log multiple times per second: 
  

  • YYYY-MM-DDTHH:MM:SS.483Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
    YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)ilo : Open could not dequeue a packet

• This message is repeated multiple times per second. Note: It is normal to see these messages every 30 seconds or so. However, more frequent entries can cause the storage to disconnect.
• ESXi host can be in a hung or unresponsive state.
• VMs will continue to run in host memory but eventually they will show as invalid.
• The below error may be seen in the environment as well: 

  • 503 Service Unavailable (Failed to connect to endpoint:.. 

• The configured scratch location can be wiped and set back to the default.

Environment

• VMware vSphere 8.0.X
• VMware vSphere 7.0.X

Cause

• As per HPE documentation, HPE WBEM Provider will try to open a channel with HPE Integrated Lights-Out (iLO) every 30 seconds.
• It is a normal message that will also be seen with HPE Agentless Management Service (AMS), and HPE Smart Update Tool (iSUT) when trying to find an unused hpilo channel.
• It will be seen in all VMware ESXi versions, e.g, ESXi 6.5, 6.7 and 7.x that run the native versions of the hpilo driver.
• The message is logged by the operating system in the vmkernel.log when the ilo driver fails to open a command for a channel that is already in use. Read more at HPE Support Center
• In this case, the frequency of logging is higher than expected i.e. excessive entries per second (3 per channel, 15 channels in total).
• ESXi - /var/log/vmkernel.log
  

  YYYY-MM-DDTHH:MM:SS.483Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)ilo : Open could not dequeue a packet
  
  YYYY-MM-DDTHH:MM:SS.496Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.509Z cpu51:2108940)ilo : Open could not dequeue a packet
  
  YYYY-MM-DDTHH:MM:SS.509Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb##'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
  YYYY-MM-DDTHH:MM:SS.521Z cpu51:2108940)ilo : Open could not dequeue a packet

• Note: By design, logging of 15 iLO channels numerates in chorological order. viz hpilo-d0ccb01, hpilo-d0ccb02. till hpilo-d0ccb15. The probe repeats once one cycle of all 15 channels completes.
• Generally these messages can be ignored unless it is repeating multiple times per second. This will clog up I/O causing storage disconnections and VM latency or failures. 

Resolution

• Update HPE drivers and firmware to the latest compatible version and reboot the host.
• For more information on this issue, it is recommended to contact HPE hardware support.