Tracking High-Volume Logs at the Source: Monitoring syslog-agent Throughput in Tanzu

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

In massive cloud platforms like VMware Tanzu, a single misconfigured application or log loop can quickly flood your log infrastructure, leading to dropped messages, saturated network queues, and inflated storage costs. When troubleshooting a massive influx of logs, waiting for downstream aggregation pipelines to parse the data can take too long.

The most precise way to find your "noisy neighbors" is to measure log velocity right at the source by auditing the syslog-agent or fluentd processes running inside your virtual machines (VMs).

Environment

• Platform: VMware Tanzu Application Service (TAS) / Cloud Foundry (CF)

• Management Tools: BOSH CLI, Ops Manager

• Operating System: Ubuntu Stemcells (Linux-based)

• Target Processes: syslog-agent, rsyslogd, or fluentd

Cause

Every VM in a Tanzu environment utilizes a local logging daemon to tail files, listen to local domain sockets, and forward data out to your central endpoints. Because these daemons run as standard Linux processes, the operating system tracks their exact I/O profile via the /proc filesystem.

Specifically, the rchar (characters read) metric—found inside /proc/[PID]/io—tracks the cumulative number of bytes a process has passed through read-like system calls. For a syslog-agent, rchar represents the gross volume of log data the agent has processed since it started up.

Resolution

Using BOSH, platform operators can run a distributed command to check the syslog-agent's accumulated read metrics across an entire deployment simultaneously.

1.) Execute the Query: Run the following command from your Ops Manager director to pool rchar metrics across your deployment:

ubuntu@examplehost:~$ date; bosh -d cf-042f#### ssh -c 'sudo cat /proc/$(pgrep syslog-agent)/io | grep rchar' 2>/dev/null | grep " rchar" | tr -d '\r' | awk -F'|' '{print $2 " | " $1}'

2.) Analyze the Output: The command produces an instant snapshot of the log volumes being swallowed by the logging agents across different components:

 rchar: 49882725 | nats/ab8acc40-####: stdout
 rchar: 51944079 | mysql_proxy/63610b6b-####: stdout
 rchar: 203189052 | diego_brain/7f5e7c4e-####: stdout
 rchar: 347372044 | uaa/692b46e6-####: stdout
 rchar: 128427044 | mysql/020d9dfa-####: stdout
 rchar: 51180780 | cloud_controller_worker/7c88497e-####: stdout
 rchar: 98972957 | doppler/28fca4e3-####: stdout
 rchar: 2902964079 | clock_global/ef8e58fc-####: stdout
 rchar: 51108694 | nfs_server/b6ff59ef-####: stdout
 rchar: 337924315 | router/e8e9db1a-####: stdout
 rchar: 728849809 | diego_cell/3dfef0d5-####: stdout
 rchar: 66022386 | loggregator_trafficcontroller/815c574f-####: stdout
 rchar: 50558281 | mysql_monitor/80b44d77-####: stdout
 rchar: 416075388 | diego_cell/65989c20-####: stdout
 rchar: 240890048 | cloud_controller/4278a6bd-####: stdout
 rchar: 49827459 | credhub/9989101c-####: stdout
 rchar: 355436455 | diego_database/55060162-####: stdout
 rchar: 101977388 | log_cache/2c8b367f-####: stdout

3.) Calculating Real-Time Log Velocity

Because rchar is a counter that continually ticks upward from the moment a process spawns, a single snapshot only shows lifetime statistics. To understand the current velocity of the log traffic, you need to calculate a delta.

⏱️ The 60-Second Delta Rule:

Run the BOSH snippet above to establish your baseline.

Wait exactly 60 seconds.

Run the snippet a second time.

Subtracting the first rchar value from the second rchar value reveals the precise volume of data the syslog-agent processed during that one-minute window.