Application Container Unable to Resolve DNS in Google Cloud Platform

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Symptoms:

After installing the PCF, the apps manager is able to log in but no information retrieved. Further investigation found application container fail to resolve Domain Name Server (DNS) from 169.254.169.254.

 

Environment

Cause

In Elastic Runtime, the application container inherits its DNS configuration from BOSH network configuration. The "Installing Ops Manager" documentation suggests that you configure the BOSH Director's DNS as `169.254.169.254, 8.8.8.8`. However, if you omit the second DNS entry, this will configure the Director and all Bosh Deployed virtual machines (VMs) to only use GCP's internal DNS.
 

This effectively gives BOSH deployed VMs a `/etc/resolv.conf` with the following:

nameserver 127.0.0.1
nameserver 169.254.169.254

127.0.0.1 is added by the consul job running on the Diego Cell VM as part of the service discovery function via DNS. 169.254.169.254 is GCP's metadata endpoint and also functions as an internal DNS resolver.
 

As mentioned above, Elastic Runtime and your application containers inherit their DNS configuration from the Bosh Director's DNS configuration. If the Bosh Director's DNS is configured as suggested by the documentation, you'll end up with `169.254.169.254, 8.8.8.8`. If you omit `8.8.8.8`, you'll end up with just `169.254.169.254`. The second situation, omitting `8.8.8.8` (or some public DNS server) is what causes the issue described in this KB.
 

The reason for this is because of the Consul endpoint, `127.0.0.1` is not added to the container's DNS list, and the metadata endpoint of cloud providers like AWS, Azure and Google Cloud Platform is not accessible from inside the container. If it were accessible, the endpoint would expose sensitive environment information about your GCP environment to potentially untrusted apps running in the container. Because this endpoint is blocked and because no public DNS server was specified, the application container does not have access to a DNS resolver.

 

Resolution

A new field in Elastic Runtime -> Application Containers -> DNS Servers has been introduced in version 1.10. This value overrides DNS servers to be used in containers. The user can specify 169.254.169.254 in the BOSH DNS configuration, and then particularly specify 8.8.8.8 in the application container DNS configuration to override the default value 169.254.169.254.