Identify noisy applications with Log-Cache in Tanzu Application Service for VMs (TAS for VMs) 2.2 and above

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

An OSS tool, Noisy Neighbor Nozzle, can be used for identifying noisy applications which produce a large mount of logs. In order to use the tool, the app needs to be deployed on VMware Tanzu Application Service for VMs (TAS for VMs) and another UAA client needs to be created. 

In TAS 2.2 and above, the Log Cache component is introduced and it provides an easy approach to identify noisy applications or system components.

Environment

Product Version: 2.9

Resolution

Log Cache is a Loggregator feature that lets you filter and query app logs through a CLI plugin or API endpoints.

Cached app logs are available on demand, you do not need to stream them continuously. The cache length for single source can be configured at Ops Manager > PAS/TAS > Settings > Advanced Features > Maximum number of envelopes stored in Log Cache per source.

As noisy sources fill up the cache faster, you can count  the rate of how fast the source fills in the buffer to identify noisy sources.  
 

Option #1

curl fetches meta data from Log Cache API endpoint with a single query. The script with jq utility sorts noisy apps in ascending order in a few seconds without placing much workload on the foundation.

An admin user token should be provided with -H "Authorization: $(cf oauth-token)" header.

Please replace <SYSTEM_DOMAIN> with your foundation's system domain. 

$ curl -s -k -H "Authorization: $(cf oauth-token)" https://log-cache.<SYSTEM_DOMAIN>/api/v1/meta | jq '[.meta | to_entries[] | {source_id: .key, messages: .value.count|tonumber, seconds: (((.value.newestTimestamp|tonumber) - (.value.oldestTimestamp|tonumber))/1000000000)} | select(.seconds != 0) | {source_id: .source_id, average_per_second: (.messages/.seconds)}] | sort_by(.average_per_second)'

Below is sample output where sources (both system components and app guids) are listed in ascending order of average logs per second:

[
	...
  {
    "source_id": "cc",
    "average_per_second": 6.320112801964164
  },
  {
    "source_id": "17ae3870-db73-4cef-968b-c603972c39a5",
    "average_per_second": 9.753622217858121
  },
  {
    "source_id": "gorouter",
    "average_per_second": 11.56213731220775
  },
  {
    "source_id": "321770fe-3658-4ff1-8688-c1989706d33f",
    "average_per_second": 11.956911513817513
  },
  {
    "source_id": "536e9989-e2c4-44ee-a455-4df613ac3329",
    "average_per_second": 12.803792565321787
  },
  {
    "source_id": "d9307ef4-60b5-41e9-be0a-895873c90501",
    "average_per_second": 19.607830256555943
  }
]

Option #2

Query Log Cache endpoint with Log Cache CLI, refer to the following documentation: Example Uses of the Log Cache CLI Plugin

$ cf install-plugin -r CF-Community "log-cache"
$ cf log-meta --guid

In the sample output, the:

• Source ID is either the app GUID or system component name. 
• Source is either the app name or system component name. 
• Source Type: unknown indicates the app has been deleted from the foundation. Count is envelops currently in Log Cache buffer. 
• Cache Duration is the time of how long it took to produce Count envelops in Log Cache buffer.
• IMaximum number of envelopes is configured as 100000. Therefore Count is never more than 100000, because older envelopes are dropped. 

By using Count and Cache Duration, noisy sources can be identified - the shorter time the buffer is filled up, the nosier the source would be:

Source ID                             Source                                Source Type  Count   Expired  Cache Duration
c2dd9a8f-28f4-4cbf-8408-0548c8573c66  apps-manager-js-blue                  application  82142   79749    21h54m48s
8b3798cd-7a2b-40f6-9605-b7cae998b855  notifications-ui                      application  21040   23694    21h54m47s

...

8b3749e0-6a7e-42ad-b4be-d52122a58d81  php1                                  application  10520   11852    21h54m44s
rep                                   rep                                   platform     100000  2210967  7h11m9s
reverse_log_proxy                     reverse_log_proxy                     platform     9205    10367    21h54m0s
97862ea9-a2a1-438c-b14b-61f208ba5ea4  ui-health-check                       application  10520   11900    21h54m44s
00ecdc3f-c205-491d-a13f-7ea1ab789188  00ecdc3f-c205-491d-a13f-7ea1ab789188  unknown      70      0        46s
018a3099-bc3d-4c36-b84a-06e5bb2cb534  018a3099-bc3d-4c36-b84a-06e5bb2cb534  unknown      67      0        45s
047a6755-d9ae-48a8-8f76-ae271cde0a43  047a6755-d9ae-48a8-8f76-ae271cde0a43  unknown      71      0        50s