HPE iLO RESTful Interface Tool consumes excessive space in root (/) directory on ESXi hosts with external persistent storage

Products

• VMware vSphere ESXi

Issue/Introduction

• When investigating vCenter or ESXi host warnings and alerts for high space utilization for the root partition, one or more large iLORest.log files are found consuming excessive space in the Root (/) directory.
• When executing the ilorest tool on an ESXi host, you may also encounter the error OSError: [Errno 28] No space left on device.
• Example of command execution:

# export TMP=/usr/local/tmpdir
# /opt/ilorest/bin/ilorest.sh --nologo
iLOrest > login
Discovering data...Done
ERROR : Exception occurred while executing command.
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/ilorest/bin/rdmc.py", line 715, in cmdloop
    self.retcode = self._run_command(opts, nargv, help_disp=False)
  File "/opt/ilorest/bin/rdmc.py", line 435, in _run_command
    return cmd.run(args[1:], help_disp=help_disp) if len(args) > 1 else cmd.run([], help_disp=help_disp)
  File "/opt/ilorest/bin/extensions/COMMANDS/LoginCommand.py", line 108, in run
    self.loginfunction(line)
  File "/opt/ilorest/bin/extensions/COMMANDS/LoginCommand.py", line 288, in loginfunction
    self.perform_login(options, skipbuild, user_ca_cert_data, args, app_obj)
  File "/opt/ilorest/bin/extensions/COMMANDS/LoginCommand.py", line 126, in perform_login
    self.rdmc.app.login(
  File "/opt/ilorest/bin/redfish/ris/rmc.py", line 307, in login
    self.save()
  File "/opt/ilorest/bin/redfish/ris/rmc.py", line 203, in save
    self._cm.cache_rmc()
  File "/opt/ilorest/bin/redfish/ris/rmc_helper.py", line 420, in cache_rmc
    indexfh.close()
OSError: [Errno 28] No space left on device
Error accessing the file path. Verify the file path is correct and you have proper permissions.
iLOrest > exit
Logging session out.

Environment

• ESXi 7.X
• ESXi 8.X

Cause

ESXi hosts installed on HPE hardware are equipped to run a RESTful Management Interface tool, which enables communication between the Hypervisor and iLO. 
Part of this deployment is the ilorest.sh script, which generates the ilorest.log file in the same directory where the script is executed. 

If the script is run outside of it's default location and in a directory such at root (/), the logging generated can consume excessive space and cause alerts on the host or within vCenter, and can impact ESXi host performance. 

Resolution

Per HPE support, this issue is resolved in SPP 2025.09.00.00. Engage HPE for resolution and further clarifications.

Workaround:

Remove or relocate the iLORest.log file from the root directory on the ESXi host.

1. Access the ESXi Shell
2. Execute the following command to log out of any active iLO sessions:
   esxcli ilorest cmd -q logout
3. Use the commands below to delete or move the log files:
   • To delete the log file:
     rm /iLORest.log
   • To move it to the /tmp directory:
     mv /iLORest.log /tmp/
4. Verify the space held by the associated log files has been released.
   vdf -h 

Note: Always ensure that ilorest.sh is run from it's default directory, or a directory with sufficient available space, such as /tmp, to avoid unintentional space consumption in system-critical partitions.

In some situations, the ilorest script can fail to complete. If this occurs, the directory will not show any freed space after the logs are deleted.

1. Check the ESXi host for multiple instances of the script running simultaneously.
2. The following command can be used to verify: lsof | grep ilorest
   (See the example image below)
   
3. These processes can be safely killed with the following command after updating the PID value:
   kill <PID>
4. Once stopped, verify the space held by the associated log files has been released.
   vdf -h 

Additional Information

Reference: Using the RESTful Interface Tool

As per the following HPE link, a new global iLORest optional argument to omit the creation of an info log file (iLORest.log) is available in version 6.3.0.0.