HPE iLO RESTful Interface Tool Consumes Excessive Space in Root (/) Directory on ESXi Hosts with External Persistent Storage

Products

• VMware vSphere ESXi

Issue/Introduction

When investigating vCenter or ESXi host warnings and alerts for high space utilization for the root partition, one or more large ilorest.log files are found consuming excessive space in the Root (/) directory.

Environment

ESXi 7.X
ESXi 8.X

Cause

ESXi hosts installed on HPE hardware are equipped to run a RESTful Management Interface tool, which enables communication between the Hypervisor and iLO. 
Part of this deployment is the ilorest.sh script, which generates the ilorest.log file in the same directory where the script is executed. 

If the script is run outside of it's default location and in a directory such at root (/), the logging generated can consume excessive space and cause alerts on the host or within vCenter, and can impact ESXi host performance. 

Resolution

See the instructions below to remove or relocate the log

- Access the ESXi Shell
- Execute the following command to log out of any active iLO sessions:: esxcli ilorest cmd -q logout
- Use the commands below to delete or move the log files:

• To delete the log file: rm /iLOREST.log
• To move it to the /tmp directory: mv /iLOREST.log /tmp/

Note: Always ensure that ilorest.sh is run from it's default directory, or a directory with sufficient available space, such as /tmp, to avoid unintentional space consumption in system-critical partitions.

In some situations, the ilorest script can fail to complete.
- If this occurs, the directory will not show any freed space after the logs are deleted.
- Check the ESXi host for multiple instances of the script running simultaneously.
- The following command can be used to verify: lsof | grep ilorest
(See the example image below)

- These processes can be safely killed with the following command after updating the PID value: kill <PID>
- Once stopped, the space held by the associated log files will be released. 

Additional Information

Reference: Using the RESTful Interface Tool