/storage/log Partition on vCenter Filling Due to Remote ESXi Syslog Ingestion

Products

• VMware vCenter Server

Issue/Introduction

• The /storage/log partition in the vCenter Server Appliance (VCSA) reaches 100% capacity or exhausts available Inodes.
• vCenter services (e.g., trustmanagement, sps, vsan-health, pschealth) fail to start.
• The vSphere Client UI is unavailable, often displaying a "no healthy upstream" error.
• Disk usage analysis reveals large log volumes or thousands of small files under /storage/log/vmware/esx/.
• The VCSA is observed listening for syslog traffic on TCP/UDP port 514.

Environment

vCenter Server

Cause

ESXi hosts in the environment are configured to forward their syslog messages to the VCSA's IP address or FQDN. The VCSA includes a basic syslog collector intended for internal appliance services; however, it is not designed to function as a production-grade remote syslog collector for external hosts. Incoming host logs are stored in the limited /storage/log partition, where they compete for space with critical vCenter logs. This leads to rapid disk or Inode exhaustion as the VCSA lacks built-in rotation or retention mechanisms for these external log streams.

Note: This condition arises if the vCenter Server inherits an IP address previously used by a decommissioned syslog server. If the ESXi hosts were not updated to stop sending logs to that IP, they will continue attempting to forward syslog data to the vCenter Server.

Resolution

To permanently resolve this issue, stop the remote syslog ingestion at the source and clear the accumulated logs on the VCSA.

Step 1: Reconfigure ESXi Hosts

Ensure that no ESXi hosts are pointing to the VCSA for syslog collection.

1. Log in to the ESXi Host Client or use PowerCLI/Host Profiles.
2. Navigate to Host > Manage > System > Advanced System Settings.
3. Locate the Syslog.global.logHost setting.
4. Remove the VCSA's FQDN or IP address. Point this to a dedicated syslog collector (e.g., VMware Aria Operations for Logs) or leave it blank if no remote collector is used.

Step 2: Reclaim Storage on VCSA

1. Access the VCSA via SSH as root.
2. Verify disk and Inode usage:
   
   df -h /storage/log
   
   
3. Delete the accumulated remote host logs:
   
   rm -rf /storage/log/vmware/esx/*
   
   
4. Restart vCenter services:
   
   service-control --start --all