BOSH DNS or external DNS resolution fails because of wrong or unwanted DNS servers in /etc/resolv.conf in Diego Cells

Products

• Operations Manager

Issue/Introduction

BOSH DNS resolution fails due to wrong or unwanted DNS servers in /etc/resolv.conf in Diego Cells. This issue is being observed when AWS or GCP are used as IaaS. 

AWS, Azure, and GCP have a feature where a user can configure the DNS server for their network. The IaaS then injects the DNS server address to /etc/resolv.conf. If this DNS server is injected on top of the list or is not valid, BOSH DNS, or a normal DNS resolution fails.

Environment

Product Version: 2.8

Resolution

Possible resolutions are:

1. Find out which DNS server is causing the problem by querying each one of them using dig or nslookup. Go to the BOSH Tile network configuration and make sure the offending server is not specified for any of the networks. 
2. Remove the invalid DNS server from your IaaS configuration so it will not get injected into the Diego Cells any more (make sure this DNS server is not used for internal IaaS resolution or else removing it may lead to outages).
3. Refer to the following documentation to add the address to BOSH DNS Excluded Recursers: https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/tanzu-operations-manager/3-0/tanzu-ops-manager/gcp-config-manual.html

For more information about the behavior and workflow of BOSH DNS, refer to the following article: https://knowledge.broadcom.com/external/article/293624/bosh-dns-introduced-container-dns-behavi.html

Note: If you are unsure of the steps above, please open a support request for assistance, referencing this article.