Esxi goes unresponsive and Vmkernel logs are spewed with "Failed to open file 'hpilo-d0ccbXX' messages

Products

• VMware vSphere ESXi 6.0
• VMware vSphere ESXi 7.0

Issue/Introduction

      Symptoms:

• ESXi host goes in hung or unresponsive state.
• VMs will continue to run but shows up as invalid
  
  
  

Environment

vSphere ESXi 6.x

vSphere ESXi 7.x

HP Servers with HP ILO

Cause

As per the following HPE document, HPE WBEM Provider will try to open a channel with HPE Integrated Lights-Out (iLO) every 30 seconds.

https://support.hpe.com/hpesc/public/docDisplay?docId=a00029796en_us&docLocale=en_US 

However, here the frequency of logging is higher than expected i.e. around 45 log entries per second (3 per channel, 15 channels in total).

• /var/log/vmkernel.log
  2025-02-08T03:33:10.483Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb10'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
2025-02-08T03:33:10.496Z cpu51:2108940)ilo : Open could not dequeue a packet

2025-02-08T03:33:10.496Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb11'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
2025-02-08T03:33:10.509Z cpu51:2108940)ilo : Open could not dequeue a packet

2025-02-08T03:33:10.509Z cpu51:2108940)FSS: 7441: Failed to open file 'hpilo-d0ccb12'; Requested flags 0x5, world: 2108940 [sut], (Existing flags 0x0, world: 0 [unknown]): Busy
2025-02-08T03:33:10.521Z cpu51:2108940)ilo : Open could not dequeue a packet

Note: By design, logging of 15 iLO channels numerates in chorological order. viz hpilo-d0ccb01, hpilo-d0ccb02. till hpilo-d0ccb15. The probe repeats once one cycle of all 15 channels completes.

Resolution

 

• For a permanent resolution, it is recommended to contact HPE hardware support.
  
  

  Workaround:

• Reboot the ESXi host to temporarily resolve the issue.