/storage/log running full in VCSA when using it as a remote syslog collector for the ESXi hosts in the environment

Products

• VMware vCenter Server

Issue/Introduction

• vCenter is suddenly unavailable
• When attempting to open vSphere Client, instead an error message "no healthy upstream" is being shown in the browser
• While troubleshooting the issue, you notice that the /storage/log partition has run full
• Large amounts of data (if not the majority) are used by files in /storage/log/vmware/esx or /storage/log/remote, where there are multiple folders named after management IPs of FQDNs of hosts in the environment

Cause

The syslog collector in vCenter is a very basic implementation with no advanced configuration options or means to actually manage the received logs.

To store the received logs, the syslog collector uses the /storage/log partition of the VCSA, where space is limited.

With each ESXi producing logs of between 2 to 3 figure megabytes each day, in larger environment this will quickly exhaust the available storage space in /storage/log, since the syslog collector uses this partition to store  the incoming log streams, and in this partition is competing with the vCenter itself, which uses the same partition for its logs as well.

 

Resolution

1. Remove the vCenter FQDN (or address) from the remote syslog configuration on each ESXi host in the environment.
   
   
2. For detailed instructions, refer to the "Configuring Syslog on ESXi" guide.
   
   
3. After updating the configuration, navigate to /storage/log/vmware/esx and move all files from this directory. You can temporarily relocate the files to /storage/core.

 

 

Additional Information

As a support engineer  we will not initiate any delete operations. If the customer wishes to delete the files, they must execute the command themselves.

If you need a remote syslog collector, use an external product instead, such as VMware Aria Operations for Logs 8.18 or any of the available 3rd party solutions on the market.