bosh-dns performs many failed DNS lookups for removed syslog drain

Products

• VMware Tanzu Application Service

Issue/Introduction

bosh-dns does not perform any negative caching as mentioned in article BOSH DNS caching behavior explained.

In the event that a syslog drain has been removed, loggregator-agent on all BOSH deployed VM's may potentially spam and overload DNS servers with lookups.

Resolution

Newer versions of loggregator-agent cache failed lookup results in order to protect against spamming of DNS server. See release notes: https://github.com/cloudfoundry/loggregator-agent-release/releases/tag/v8.1.3

Cache failed dns lookups for 120 seconds which reduced load on DNS system when invalid drains are present

This fix is included in TAS 6.0.6 and 4.0.26. This issue is addressed in v.7.7.9 and v8.13 of loggregator-agent release included with Tanzu Application Service.

Additionally, service tiles also may include loggregator-agent release. You can check loggregator-agent version used by a deployment by command: 

bosh -d <deployment name> manifest | grep -A 1 "releases/loggregator-agent" 

url: file:///var/tempest/releases/loggregator-agent-7.7.3-ubuntu-jammy-1.318.tgz 

version: 7.7.3

It may be necessary to check release notes of service tile to see if a newer release is available with bumped loggregator-agent release. If no release contains an updated loggregator-agen then you could manually update this package by steps in KB: How to Manually Upgrade a Release