Container metrics gathering via external syslog tool

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Container metrics are generated by rep process and send directly from diego cell into the syslog. When using tools such as cf drain, it is draining the logs and metrics into chosen syslog endpoint. In some cases it may be missing container metrics due to the fact that metrics are coming enclosed in the tag field as such

798 <14>1 2020-11-11T11:39:25.60401+00:00 system.system.hello-pcf 657ed151-86d5-4359-a000-aa2890511d7a [0] - [gauge@47450 name="memory" value="3.0106841e+07" unit="bytes"][tags@47450 app_id="657ed151-86d5-4359-a000-aa2890511d7a" app_name="hello-pcf" deployment="cf-64546b78dbf6a1e82394" index="de126ac8-563d-4d1a-a23e-0d0ee988b89a" instance_id="0" ip="10.213.34.153" job="diego_cell" organization_id="d7ca1a63-78a4-415b-95cb-1b6905278c8b" organization_name="system" origin="rep" process_id="657ed151-86d5-4359-a000-aa2890511d7a" process_instance_id="ed0818f7-5d8b-43df-7685-96a2" process_type="web" product="Pivotal Application Service" source_id="657ed151-86d5-4359-a000-aa2890511d7a" space_id="e33952b6-46c6-4965-bd41-c9c5423c72b7" space_name="system" system_domain="run-05.slot-##.####.###.com"]

The logs are coming after tags:

849 <14>1 2020-11-11T11:39:34.152144+00:00 system.system.hello-pcf 657ed151-86d5-4359-a000-aa2890511d7a [CELL/0] - [tags@47450 app_id="657ed151-86d5-4359-a000-aa2890511d7a" app_name="hello-pcf" deployment="cf-64546b78dbf6a1e82394" index="de126ac8-563d-4d1a-a23e-0d0ee988b89a" instance_id="0" ip="10.213.34.153" job="diego_cell" organization_id="d7ca1a63-78a4-415b-95cb-1b6905278c8b" organization_name="system" origin="rep" process_id="657ed151-86d5-4359-a000-aa2890511d7a" process_instance_id="ed0818f7-5d8b-43df-7685-96a2" process_type="web" product="Pivotal Application Service" source_id="657ed151-86d5-4359-a000-aa2890511d7a" source_type="CELL" space_id="e33952b6-46c6-4965-bd41-c9c5423c72b7" space_name="system" system_domain="run-05.slot-##.###.#####.com"] Cell de126ac8-563d-4d1a-a23e-0d0ee988b89a stopping instance ed0818f7-5d8b-43df-7685-96a2

Tags are used for query purposes. 

Environment

Product Version: 2.8

Resolution

When setting up a cf drain (https://docs.cloudfoundry.org/devguide/services/log-management.html) and not receiving metrics but receiving logs, it is recommended to review the log management tool used and to check the filtering on it, as the metrics come in tags field, but logs coming after the tags field. Sometimes this may cause an inconsistency where the logs are showing but not metrics being printed due to them being inclosed in tags.