Doppler fails to connect to Log Cache on port 6067 in Tanzu Application Service for VMs 2.8+

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

In the release of Tanzu Application Service (TAS) for VMs 2.8+, there was feature introduced to allow Log Cache to consume log messages through Syslog protocol.

A Syslog drain is configured for each of the syslog agents running on the VMs. The Syslog drain is added in every VM with the syslog agent since it is configured by default in the TAS manifest.

See the following file for more information: log-cache-syslog-server.yml

In the manifest, you'll see there is an aggregate syslog drain which is configured to point to doppler, where the Log Cache job resides.

Whilst the syslog drain pointing to doppler is configured by default in your manifest, the ingestion and enablement of Log Cache ingesting logs and metrics over Syslog is disabled by default. The drain was added so platform operators do not need to configure the drain for Log Cache ingestion over Syslog if they decide to enable Log Cache ingestion over Syslog.

If this feature is disabled, then you will see the following error in /var/vcap/sys/log/loggr-syslog-agent/loggr-syslog-agent.stderr.log:

2020/12/04 20:32:03 failed to write to doppler.service.cf.internal:6067, retrying in 512ms, err: dial tcp xxxxxx:6067: connect: connection refused
2020/12/04 20:32:04 failed to write to doppler.service.cf.internal:6067, retrying in 1.024s, err: dial tcp xxxxxx:6067: connect: connection refused
2020/12/04 20:32:05 failed to write to doppler.service.cf.internal:6067, retrying in 2.048s, err: dial tcp xxxxxxxx:6067: connect: connection refused

Environment

Product Version: 2.8

Resolution

If you are on TAS 2.8+, then you can either ignore the messages in the loggr-syslog-agent logs as they are configured to emit every 15 seconds.

Note: An internal issue has been created to reduce the frequency of these error messages when this feature is disabled.

To resolve this error, you can toggle the checkbox for "Enable Log Cache syslog ingestion" in TAS > System Logging to enable ingestion on the Syslog drain. Enabling the drain would open port 6067 on doppler and resolve the error you are seeing.

If you do not want to enable the feature and are ingesting Platform Logs over Syslog and the error logs are consuming space in your external Syslog service, then you can prevent the logs from being ingested over Syslog. To do this, the following filter to be added in your TAS/PAS Tile > System Logging > Custom rsyslog configuration:

if ($app-name == "loggr-syslog-agent" and $msg contains "failed to write to doppler.service.cf.internal:6067") then stop