What kind of extra metadata will be included by enabling "Default loggregator drain metadata" option

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

A new feature Reduce Traffic to Syslog Drains was introduced in TAS 2.12. 
The same function was also available from TAS 2.9.23, 2.10.15, 2.11.3 with following description in release note.
[Bug Fix] Add an option to remove extra metadata from syslog drains

The feature can be enabled/disabled on System Logging page of TAS Settings as illustrated in image below.

More details are available in TAS doc. 

Sometimes users might want to know what kind of extra metadata will be added/removed by enabling/disabling this option. This article provides sample messages for reference as well as how to check them via rsyslogd.

Resolution

Sample message when Default loggregator drain metadata checkbox is selected

2021-10-25T01:36:31.276533+00:00 system.system.testapp-metadata 8f1777ac-ed04-4415-9ed3-17eb5ed0b763 [APP/PROC/WEB/0] - [tags@47450 app_id="8f1777ac-ed04-4415-9ed3-17eb5ed0b763" app_name="testapp-metadata" deployment="cf-3dc000e731978fd662d0" index="0fc93f04-e98c-4ae0-86e8-83a4fd31b2ea" instance_id="0" ip="10.213.41.166" job="diego_cell" organization_id="f0d812fe-725a-40fa-b3c3-e03ad15588ec" organization_name="system" origin="rep" process_id="8f1777ac-ed04-4415-9ed3-17eb5ed0b763" process_instance_id="18f33859-db8c-41a5-62ed-c195" process_type="web" product="VMware Tanzu Application Service" source_id="8f1777ac-ed04-4415-9ed3-17eb5ed0b763" source_type="APP/PROC/WEB" space_id="4c6361a2-f7c6-4dca-ba58-07fc6bb8baf7" space_name="system" system_domain="run-19.vmware.com"] test-app. Says Hello. on index: 0  

Sample message when Default loggregator drain metadata checkbox is deselected

2021-10-25T01:51:36.276421+00:00 system.system.testapp-metadata 8f1777ac-ed04-4415-9ed3-17eb5ed0b763 [APP/PROC/WEB/0] - - test-app. Says Hello. on index: 0 

Sometimes rsyslogd will be used by user as the remote syslog drain destination. However no extra medata is included in message received by rsyslogd even Default loggregator drain metadata checkbox is selected. This is because by default rsyslogd is using the older RFC3164 format to parse the message. But TAS uses the newer 5424 format to encode the message, so extra metadata are missing in messages which are written into /var/log/syslog by rsyslogd. 
In order to make rsyslogd to write all data it receives from remote, you can configure rsyslogd to output raw message instead of default format. Adding a new template with type of rawmsg to /etc/rsyslog.conf and set it as the default template to output received messages. 

template(name="SyslogStorerTemplate" type="list") {
  property(name="rawmsg")
  constant(value="\n")
}
$ActionFileDefaultTemplate SyslogStorerTemplate 

Don't forget to restart rsyslogd with either service ryslogd restart or systemctl restart rsyslogd to make the change effect.