BOSH DNS introduced Container DNS Behavior Changes in PAS 2.0
Article ID: 293624
Updated On:
Products
Issue/Introduction
BOSH DNS was introduced in PAS 2.0. This article describes how BOSH DNS changes "how containers perform DNS lookups on a Diego Cell".
Environment
Resolution
Container Life before bosh DNS Before PAS 2.0
The configured DNS servers (located in the create networks tab under the director tile) will get inserted into the Diego Cells /etc/resolv.conf file by the bosh agent during deployment. For example, if we have DNS server "10.#.#.#,10.#.#.3" configured in the ops manager, the file would look like below:
nameserver 127.#.#.# nameserver 10.#.#.# nameserver 10.#.#.#
127.#.#.# is the consul agent running on the cell and is mainly used for the service discovery of internal PAS services. However, consul agent will perform a recursive DNS lookup using the configured external DNS server when the search domain is not in services.cf.internal.
When a Container gets created, Diego will populate the /etc/resolv.conf with the external DNS server as followed:
nameserver 10.#.#.# nameserver 10.#.#.#
Now all DNS lookups by the container will be serviced by the external DNS servers as expected.
Container Life with BOSH DNS
With the introduction of BOSH DNS, the Diego cell will have a listening process on internal IP address 169.254.0.2:
tcp 0 0 169.###.#.#:53 0.0.0.0:* LISTEN 5715/bosh-dns
With the same dns server configuration as the previous example the Diego Cell will have the following configuration in /etc/resolv.conf:
nameserver 127.#.#.# nameserver 10.#.#.# nameserver 10.#.#.#
This means the DNS lookup behavior for the Diego Cell does not change at all.
The APP Container running on the diego cell will now have this configuration in its /etc/resolv.conf
nameserver 169.###.#.2
All DNS lookups will now go through the BOSH-DNS process running on the diego cell.
How BOSH DNS behaves
Startup of the BOSH-DNS process:
When BOSH DNS starts up, it randomly selects one of the externally configured IP addresses in /etc/resolv.conf. In the case of windows, BOSH DNS will query all the network adapter settings for IP addresses. The following powershell script can be used to identify what servers it will find:
try {
[array]$routeable_interfaces = Get-WmiObject Win32_NetworkAdapterConfiguration | Where { $_.IpAddress -AND ($_.IpAddress | Where { $addr = [Net.IPAddress] $_; $addr.AddressFamily -eq "InterNetwork" -AND ($addr.address -BAND ([Net.IPAddress] "255.255.0.0").address) -ne ([Net.IPAddress] "169.###.#.0").address }) }
$interface = (Get-WmiObject Win32_NetworkAdapter | Where { $_.DeviceID -eq $routeable_interfaces[0].Index }).netconnectionid
(Get-DnsClientServerAddress -InterfaceAlias $interface -AddressFamily ipv4 -ErrorAction Stop).ServerAddresses
} catch {
$Host.UI.WriteErrorLine($_.Exception.Message)
Exit 1
}
Exit 0
In both the cases of Linux and Windows, the loopback interface of 127.0.0.1 is ignored. To find out which IP is used, you can look at its log file and observe this message during startup. Also when recursive queries are initiated BOSH DNS will log which server is used to perform the lookup through:
[FailoverRecursor] 2018/02/28 22:38:58 INFO - starting preference: 10.#.#.#:53
This behavior is different from how linux behaves. For example, when you run programs like the host, nslookup, dig, etc.. they will likely be calling a common function such as getaddrinfo that does things like read /etc/nsswitch.conf and decides how to perform the resolution. These commands will certainly read the /etc/resolv.conf from top to bottom resulting in 127.0.0.1 getting picked first every time in case of perfoming a lookup from the Diego cell. It is important to understand this difference in behavior when you are troubleshooting APP Container lookup issues
Limitations
BOSH DNS does not support reverse DNS lookups in early PAS 2.0 release. This is fixed in BOSH-DNS release v0.1.3 and v1.3.0. Refer to the PAS release notes to identify which versions have this feature.
Additional Information
A full description of BOSH DNS is here in the Open Source Documentation https://bosh.io/docs/dns.html.
Amazon Web Services and Google Cloud Platform will inject a DNS server defined in the subnet into /etc/resolv.conf. This entry is typically injected as the last nameserver in /etc/resolv.conf. This could result in BOSH DNS selecting the IAAS DNS server on startup which may cause undesired DNS lookup behavior. Make sure to disable this DNS feature in your IAAS.
Operations Manager will allow Operators to add the IAAS DNS servers to an exclude list which sets the "excluded_recursors" in deployment manifests. Any DNS server in the exclude list will be ignored by BOSH DNS.
Feedback
