BOSH DNS caching behavior explained

Products

• Operations Manager

Issue/Introduction

This Knowledge Base (KB) article details the caching behavior of BOSH DNS for maintaining a cache of DNS queries.

bosh-dns utilizes a plugin called coredns for performing DNS queries. bosh-dns caching defaults to the behavior of the coredns cache plugin. The plugin is an automatically bumped dependency within the bosh-dns release.

Environment

Product Version: 2.8

Resolution

coredns utilizes TTL to determine how long to cache a result. DNS TTL (time to live) is a setting that tells the DNS resolver how long to cache a query before requesting a new one.

For positive caching, coredns respects the TTL in the recursor's response. So bosh-dns will cache results for a length of time based on configured TTL settings of remote DNS server. Note that the upstream might refresh with a new TTL before the cached TTL expires, so cached responses might be lower when comparing cached and upstream responses.

For negative caching, meaning a server cannot provide an answer for the query, bosh-dns will not cache the response for address records and most other query types. 

Options:
The coredns TTL setting is not a property that is configurable within BOSH DNS. As a result, you are not able to tune this expiration period.

The cache handler can only be enable and disabled. It is not configured further and uses the coredns defaults:

• https://github.com/cloudfoundry/bosh-dns-release/blob/master/src/bosh-dns/dns/server/handlers/cache_handler.go#L28-L30
• https://github.com/coredns/coredns/blob/master/plugin/cache/cache.go#L240-L245

If recursors are slow due to high load you might want to tweak "

recursor_timeout " vi Operations Manager -> Director -> BOSH DNS > Recursor Timeout 

Disabling Cache is only configurable in custom handlers.  You can not configure BOSH DNS default behavior to disable cache.   You can add a customer handler here Ops Manager > Director > BOSH DNS > Handlers config.   

• https://docs.pivotal.io/platform/ops-manager/2-9/vsphere/config.html#bosh-dns

For example:

{"cache":{"enabled":false}, ...}

If you have an overloaded or unresponsive DNS server in resolv.conf then you may need to add this server to  Ops Manager > Director > BOSH DNS > Exclude Recursors so it is not selected for use.

Additional Information

There is also a known bug relating with BOSH DNS related to negative caching. The problem can occur under the follow circumstances:

1. App on cell tries to look up something.hostname external and that lookup times out or fails.
2. For the next 5 seconds any app on that same cell that tries to lookup the same domain name someting.hostname 
3. It will fail immediately because bosh cached the previously failed response.
4. After 5 seconds bosh dns will recurse any subsequent lookups.

The above situation is normally only applicable when there is a single resolver. If there is is more than one resolver, then BOSH DNS will try them all before failing and caching the failed lookup.

This BOSH DNS bug is fixed by a dependency update in bosh-dns 1.21.

https://github.com/coredns/coredns/tree/master/plugin/cache#cache