Insufficient instance RAM triggers DNS storm to upstream DNS server

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

In environments with a large number of syslog drain bindings on a TPCF platform, loggr-syslog-agent job can generate an enormous volume of DNS queries - one for each drain binding at 15-second interval. On instances with only 1GB of RAM, this flood of queries can exhaust the UDP buffer. As a result, DNS responses may not be cached locally despite their TTLs, forcing bosh-dns to forward nearly every query to the upstream DNS server.

This issue could occur with any instances on platform, not only TPCF, but other tile instances when they meet the conditions: 

• loggr-syslog-agent job
• small RAM size
• large number of syslog drain bindings

Resolution

If the issue is observed on your platform, please scale up the impacted instance_group RAM size from 1GB to at least 2GB. A permanent fix with loggr-syslog-agent would be delivered in future releases.