VCSA /storage/log Directory Full Due to Remote Syslog Collector Log Files

Products

• VMware vCenter Server

Issue/Introduction

• The /storage/log partition in the vCenter Server Appliance becomes completely full with large syslog.log and syslog.backup files.
• The directory /storage/log/vmware/esx/<host‑IP-or‑FQDN> accumulates logs consistently—even after cleanup attempts.
• vCenter becomes suddenly unavailable, and users see the error “no healthy upstream” when trying to open the vSphere Client.
• Investigation reveals that disk usage is dominated by host-specific subfolders under /storage/log/vmware/esx/vCenter/

Environment

vCenter Server Appliance 7.0

Cause

• When ESXi hosts are configured to forward their syslog messages to the vCenter appliance, the built-in syslog collector stores incoming logs in /storage/log—a limited space shared with vCenter’s own logs.
• Under high log volume, this quickly exhausts available disk space.
• Specifically, in vCenter 7.0+, a known issue prevents rsyslogd from properly reading log-rotation configuration from /etc/logrotate.d/vmware-syslog.lr after syslog collector changes.
   

Resolution

Immediate Workaround:

SSH into the vCenter Server and move existing files out of /storage/log/vmware/esx/ to another partition, such as /storage/core, to immediately free space

-> Take vCenter snapshot.
 

Scripted Workaround:

1. Copy the syslogConfigFix.sh script attached to this article to the problem vCenter appliance.
2. Make the script executable by running:
   # chmod +x syslogConfigFix.sh
    
3. Execute the script by running:
   # ./syslogConfigFix.sh

For Example:

root@vcsa_name [ ~ ]# chmod +x syslogConfigFix.sh
root@vcsa_name [ ~ ]# ./syslogConfigFix.sh

 

Manual Fix:

1. Backup /etc/rsyslog.conf:
        #cp /etc/rsyslog.conf /etc/rsyslog.conf.bak
   
   
2. Edit /etc/rsyslog.conf with vi editor:

      # vi rsyslog.conf

Replace legacy line:
*.* ?esxLoc;esxFmt

with:
if ($hostname != $$myhostname) then ?esxLoc;esxFmt

3. Backup /etc/vmware-syslog/syslog.conf:
         #cp /etc/vmware-syslog/syslog.conf /etc/vmware-syslog/syslog.conf.bak
   
4. Remove the '& stop' line at the end of the syslog.conf file.
5. Restart rsyslog:
   #systemctl restart rsyslog

• After applying a manual or scripted fix, logs will be written to both /var/log/* and to the external syslog server—not stored in /storage/log/vmware/esx/*